Mercurial > bitten > bitten-test
annotate bitten/slave.py @ 895:7d93d6358fe0
Use our own HTTPBasicAuthHandler under Python 2.6 to avoid issue http://bugs.python.org/issue8797. Fixes #658.
author | hodgestar |
---|---|
date | Wed, 09 Mar 2011 14:48:35 +0000 |
parents | 60d0807334ec |
children | ce35ae342a66 |
rev | line source |
---|---|
379 | 1 # -*- coding: utf-8 -*- |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
2 # |
832 | 3 # Copyright (C) 2007-2010 Edgewall Software |
408
933105ab516b
Update file headers and other stuff pointing to the old home.
cmlenz
parents:
405
diff
changeset
|
4 # Copyright (C) 2005-2007 Christopher Lenz <cmlenz@gmx.de> |
163 | 5 # All rights reserved. |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
6 # |
163 | 7 # This software is licensed as described in the file COPYING, which |
8 # you should have received as part of this distribution. The terms | |
408
933105ab516b
Update file headers and other stuff pointing to the old home.
cmlenz
parents:
405
diff
changeset
|
9 # are also available at http://bitten.edgewall.org/wiki/License. |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
10 |
313 | 11 """Implementation of the build slave.""" |
12 | |
82
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
13 from datetime import datetime |
457
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
14 import errno |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
15 import urllib |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
16 import urllib2 |
15
06207499c58c
* Use logging in the BEEP core as well as in the master and slave scripts. Closes #4.
cmlenz
parents:
14
diff
changeset
|
17 import logging |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
18 import os |
66
df40a4eb7553
Use the standard {{{platform}}} module for reporting of system info by slaves. Closes #6.
cmlenz
parents:
63
diff
changeset
|
19 import platform |
90
2c4e104afef8
The build master now transmits snapshot archives without blocking while reading the file and sending the BEEP frames. Closes #17.
cmlenz
parents:
88
diff
changeset
|
20 import shutil |
457
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
21 import socket |
654
cc1fefd5d814
0.6dev: Moved a constant declaration so that master should still work with Python 2.4. Updated docs with regards to Python requirements + added a changelog message from cmlenz.
osimons
parents:
652
diff
changeset
|
22 import sys |
42
efa525876b1e
Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents:
34
diff
changeset
|
23 import tempfile |
392 | 24 import time |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
25 import re |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
26 import cookielib |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
27 import threading |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
28 import os |
835
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
29 import mimetools |
675
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
30 from ConfigParser import MissingSectionHeaderError |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
31 |
654
cc1fefd5d814
0.6dev: Moved a constant declaration so that master should still work with Python 2.4. Updated docs with regards to Python requirements + added a changelog message from cmlenz.
osimons
parents:
652
diff
changeset
|
32 from bitten import PROTOCOL_VERSION |
61
47ab019508dd
Moved {{{BuildError}}} class into package {{{bitten.build}}}.
cmlenz
parents:
60
diff
changeset
|
33 from bitten.build import BuildError |
675
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
34 from bitten.build.config import Configuration, ConfigFileNotFound |
436
cfbc9ee622d5
Finish the move of build configuration management into the admin interface.
cmlenz
parents:
424
diff
changeset
|
35 from bitten.recipe import Recipe |
392 | 36 from bitten.util import xmlio |
895
7d93d6358fe0
Use our own HTTPBasicAuthHandler under Python 2.6 to avoid issue http://bugs.python.org/issue8797. Fixes #658.
hodgestar
parents:
891
diff
changeset
|
37 from bitten.util.compat import HTTPBasicAuthHandler |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
38 |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
39 EX_OK = getattr(os, "EX_OK", 0) |
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
40 EX_UNAVAILABLE = getattr(os, "EX_UNAVAILABLE", 69) |
675
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
41 EX_IOERR = getattr(os, "EX_IOERR", 74) |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
42 EX_PROTOCOL = getattr(os, "EX_PROTOCOL", 76) |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
43 EX_NOPERM = getattr(os, "EX_NOPERM", 77) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
44 |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
45 FORM_TOKEN_RE = re.compile('__FORM_TOKEN\" value=\"(.+)\"') |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
46 |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
47 __all__ = ['BuildSlave', 'ExitSlave'] |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
48 __docformat__ = 'restructuredtext en' |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
49 |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
90
diff
changeset
|
50 log = logging.getLogger('bitten.slave') |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
90
diff
changeset
|
51 |
457
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
52 # List of network errors which are usually temporary and non critical. |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
53 temp_net_errors = [errno.ENETUNREACH, errno.ENETDOWN, errno.ETIMEDOUT, |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
54 errno.ECONNREFUSED] |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
55 |
474
8c3dfe0efe78
Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents:
466
diff
changeset
|
56 def _rmtree(root): |
549
cb137f2121b9
Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents:
545
diff
changeset
|
57 """Catch shutil.rmtree failures on Windows when files are read-only, and only remove if root exists.""" |
474
8c3dfe0efe78
Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents:
466
diff
changeset
|
58 def _handle_error(fn, path, excinfo): |
8c3dfe0efe78
Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents:
466
diff
changeset
|
59 os.chmod(path, 0666) |
8c3dfe0efe78
Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents:
466
diff
changeset
|
60 fn(path) |
549
cb137f2121b9
Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents:
545
diff
changeset
|
61 if os.path.exists(root): |
cb137f2121b9
Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents:
545
diff
changeset
|
62 return shutil.rmtree(root, onerror=_handle_error) |
cb137f2121b9
Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents:
545
diff
changeset
|
63 else: |
cb137f2121b9
Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents:
545
diff
changeset
|
64 return False |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
65 |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
66 |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
67 class SaneHTTPRequest(urllib2.Request): |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
68 |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
69 def __init__(self, method, url, data=None, headers={}): |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
70 urllib2.Request.__init__(self, url, data, headers) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
71 self.method = method |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
72 |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
73 def get_method(self): |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
74 if self.method is None: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
75 self.method = self.has_data() and 'POST' or 'GET' |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
76 return self.method |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
77 |
835
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
78 |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
79 def encode_multipart_formdata(fields): |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
80 """ |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
81 Given a dictionary field parameters, returns the HTTP request body and the |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
82 content_type (which includes the boundary string), to be used with an |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
83 httplib-like call. |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
84 |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
85 Normal key/value items are treated as regular parameters, but key/tuple |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
86 items are treated as files, where a value tuple is a (filename, data) tuple. |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
87 |
870 | 88 For example:: |
89 | |
835
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
90 fields = { |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
91 'foo': 'bar', |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
92 'foofile': ('foofile.txt', 'contents of foofile'), |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
93 } |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
94 body, content_type = encode_multipart_formdata(fields) |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
95 |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
96 Note: Adapted from http://code.google.com/p/urllib3/ (MIT license) |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
97 """ |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
98 |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
99 BOUNDARY = mimetools.choose_boundary() |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
100 ENCODE_TEMPLATE= "--%(boundary)s\r\n" \ |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
101 "Content-Disposition: form-data; name=\"%(name)s\"\r\n" \ |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
102 "\r\n%(value)s\r\n" |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
103 ENCODE_TEMPLATE_FILE = "--%(boundary)s\r\n" \ |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
104 "Content-Disposition: form-data; name=\"%(name)s\"; " \ |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
105 "filename=\"%(filename)s\"\r\n" \ |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
106 "Content-Type: %(contenttype)s\r\n" \ |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
107 "\r\n%(value)s\r\n" |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
108 |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
109 body = "" |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
110 for key, value in fields.iteritems(): |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
111 if isinstance(value, tuple): |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
112 filename, value = value |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
113 body += ENCODE_TEMPLATE_FILE % { |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
114 'boundary': BOUNDARY, |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
115 'name': str(key), |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
116 'value': str(value), |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
117 'filename': str(filename), |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
118 'contenttype': 'application/octet-stream' |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
119 } |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
120 else: |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
121 body += ENCODE_TEMPLATE % { |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
122 'boundary': BOUNDARY, |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
123 'name': str(key), |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
124 'value': str(value) |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
125 } |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
126 body += '--%s--\r\n' % BOUNDARY |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
127 content_type = 'multipart/form-data; boundary=%s' % BOUNDARY |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
128 return body, content_type |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
129 |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
130 |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
131 class KeepAliveThread(threading.Thread): |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
132 "A thread to periodically send keep-alive messages to the master" |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
133 |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
134 def __init__(self, opener, build_url, single_build, keepalive_interval): |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
135 threading.Thread.__init__(self, None, None, "KeepaliveThread") |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
136 self.build_url = build_url |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
137 self.keepalive_interval = keepalive_interval |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
138 self.single_build = single_build |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
139 self.last_keepalive = int(time.time()) |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
140 self.kill = False |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
141 self.opener = opener |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
142 |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
143 def keepalive(self): |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
144 log.debug('Sending keepalive') |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
145 method = 'POST' |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
146 url = self.build_url + '/keepalive/' |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
147 body = None |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
148 shutdown = False |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
149 headers = { |
891
60d0807334ec
Set 'Content-Length' header on keep-alive requests. Fixes #667. Thanks eblot.
osimons
parents:
889
diff
changeset
|
150 'Content-Type': 'application/x-bitten+xml', |
60d0807334ec
Set 'Content-Length' header on keep-alive requests. Fixes #667. Thanks eblot.
osimons
parents:
889
diff
changeset
|
151 'Content-Length': '0' |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
152 } |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
153 log.debug('Sending %s request to %r', method, url) |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
154 req = SaneHTTPRequest(method, url, body, headers or {}) |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
155 try: |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
156 return self.opener.open(req) |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
157 except urllib2.HTTPError, e: |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
158 # a conflict error lets us know that we've been |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
159 # invalidated. Ideally, we'd engineer something to stop any |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
160 # running steps in progress, but killing threads is tricky |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
161 # stuff. For now, we'll wait for whatever's going |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
162 # on to stop, and the main thread'll figure out that we've |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
163 # been invalidated. |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
164 log.warning('Server returned keepalive error %d: %s', e.code, e.msg) |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
165 except: |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
166 log.warning('Server returned unknown keepalive error') |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
167 |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
168 def run(self): |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
169 log.debug('Keepalive thread starting.') |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
170 while (not self.kill): |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
171 now = int(time.time()) |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
172 if (self.last_keepalive + self.keepalive_interval) < now: |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
173 self.keepalive() |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
174 self.last_keepalive = now |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
175 |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
176 time.sleep(1) |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
177 log.debug('Keepalive thread exiting.') |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
178 |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
179 def stop(self): |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
180 log.debug('Stopping keepalive thread') |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
181 self.kill = True |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
182 self.join(30) |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
183 log.debug('Keepalive thread stopped') |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
184 |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
185 |
392 | 186 class BuildSlave(object): |
542 | 187 """HTTP client implementation for the build slave.""" |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
188 |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
189 def __init__(self, urls, name=None, config=None, dry_run=False, |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
190 work_dir=None, build_dir="build_${build}", |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
191 keep_files=False, single_build=False, |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
192 poll_interval=300, keepalive_interval = 60, |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
193 username=None, password=None, |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
194 dump_reports=False, no_loop=False, form_auth=False): |
313 | 195 """Create the build slave instance. |
196 | |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
197 :param urls: a list of URLs of the build masters to connect to, or a |
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
198 single-element list containing the path to a build recipe |
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
199 file |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
200 :param name: the name with which this slave should identify itself |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
201 :param config: the path to the slave configuration file |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
202 :param dry_run: wether the build outcome should not be reported back |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
203 to the master |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
204 :param work_dir: the working directory to use for build execution |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
205 :param build_dir: the pattern to use for naming the build subdir |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
206 :param keep_files: whether files and directories created for build |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
207 execution should be kept when done |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
208 :param single_build: whether this slave should exit after completing a |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
209 single build, or continue processing builds forever |
449
01503ed994e7
Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents:
442
diff
changeset
|
210 :param poll_interval: the time in seconds to wait between requesting |
451 | 211 builds from the build master (default is five |
212 minutes) | |
870 | 213 :param keepalive_interval: the time in seconds to wait between sending |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
214 keepalive heartbeats (default is 30 seconds) |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
215 :param username: the username to use when authentication against the |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
216 build master is requested |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
217 :param password: the password to use when authentication is needed |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
218 :param dump_reports: whether report data should be written to the |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
219 standard output, in addition to being transmitted |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
220 to the build master |
525
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
221 :param no_loop: for this slave to just perform a single check, regardless |
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
222 of whether a build is done or not |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
223 :param form_auth: login using AccountManager HTML form instead of |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
224 HTTP authentication for all urls |
313 | 225 """ |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
226 self.urls = urls |
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
227 self.local = len(urls) == 1 and not urls[0].startswith('http://') \ |
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
228 and not urls[0].startswith('https://') |
392 | 229 if name is None: |
230 name = platform.node().split('.', 1)[0].lower() | |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
231 self.name = name |
392 | 232 self.config = Configuration(config) |
185
2c24d9a950ed
Add a `--dry-run` option to the build slave. This will result in the slave being registered and executing builds, but without submitting the progress and results of the build back to the server. Useful for getting the configuration of new slaves right without polluting the database with invalid builds.
cmlenz
parents:
163
diff
changeset
|
233 self.dry_run = dry_run |
208
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
234 if not work_dir: |
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
235 work_dir = tempfile.mkdtemp(prefix='bitten') |
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
236 elif not os.path.exists(work_dir): |
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
237 os.makedirs(work_dir) |
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
238 self.work_dir = work_dir |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
239 self.build_dir = build_dir |
241
d3c8a74922cd
* Add an option to `bitten-slave` that tells it to keep any files in the working directory.
cmlenz
parents:
233
diff
changeset
|
240 self.keep_files = keep_files |
377 | 241 self.single_build = single_build |
525
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
242 self.no_loop = no_loop |
449
01503ed994e7
Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents:
442
diff
changeset
|
243 self.poll_interval = poll_interval |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
244 self.keepalive_interval = keepalive_interval |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
245 self.dump_reports = dump_reports |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
246 self.cookiejar = cookielib.CookieJar() |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
247 self.username = username \ |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
248 or self.config['authentication.username'] or '' |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
249 |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
250 if not self.local: |
569
74bc339ce7d5
0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents:
560
diff
changeset
|
251 self.password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm() |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
252 if self.username: |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
253 log.debug('Enabling authentication with username %r', |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
254 self.username) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
255 self.form_auth = form_auth |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
256 password = password \ |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
257 or self.config['authentication.password'] or '' |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
258 self.config.packages.pop('authentication', None) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
259 urls = [url[:-7] for url in urls] |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
260 self.password_mgr.add_password( |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
261 None, urls, self.username, password) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
262 self.auth_map = dict(map(lambda x: (x, False), urls)) |
569
74bc339ce7d5
0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents:
560
diff
changeset
|
263 |
74bc339ce7d5
0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents:
560
diff
changeset
|
264 def _get_opener(self): |
652
de04ce69da53
0.6dev: Removing code and updated docs related to Trac < 0.11 and Python < 2.4 (base requirements for Bitten 0.6).
osimons
parents:
649
diff
changeset
|
265 opener = urllib2.build_opener(urllib2.HTTPErrorProcessor()) |
895
7d93d6358fe0
Use our own HTTPBasicAuthHandler under Python 2.6 to avoid issue http://bugs.python.org/issue8797. Fixes #658.
hodgestar
parents:
891
diff
changeset
|
266 opener.add_handler(HTTPBasicAuthHandler(self.password_mgr)) |
569
74bc339ce7d5
0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents:
560
diff
changeset
|
267 opener.add_handler(urllib2.HTTPDigestAuthHandler(self.password_mgr)) |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
268 opener.add_handler(urllib2.HTTPCookieProcessor(self.cookiejar)) |
569
74bc339ce7d5
0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents:
560
diff
changeset
|
269 return opener |
74bc339ce7d5
0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents:
560
diff
changeset
|
270 opener = property(_get_opener) |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
271 |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
272 def request(self, method, url, body=None, headers=None): |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
273 log.debug('Sending %s request to %r', method, url) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
274 req = SaneHTTPRequest(method, url, body, headers or {}) |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
275 try: |
559
bf3e0c2f6b80
Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents:
549
diff
changeset
|
276 resp = self.opener.open(req) |
bf3e0c2f6b80
Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents:
549
diff
changeset
|
277 if not hasattr(resp, 'code'): |
bf3e0c2f6b80
Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents:
549
diff
changeset
|
278 resp.code = 200 |
bf3e0c2f6b80
Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents:
549
diff
changeset
|
279 return resp |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
280 except urllib2.HTTPError, e: |
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
281 if e.code >= 300: |
647
6fe6fc8ee36c
0.6dev: Follow-up to [719]. Seems some http errors don't populate `headers` (at least apache digest auth challenge didn't).
osimons
parents:
645
diff
changeset
|
282 if hasattr(e, 'headers') and \ |
6fe6fc8ee36c
0.6dev: Follow-up to [719]. Seems some http errors don't populate `headers` (at least apache digest auth challenge didn't).
osimons
parents:
645
diff
changeset
|
283 e.headers.getheader('Content-Type', '' |
6fe6fc8ee36c
0.6dev: Follow-up to [719]. Seems some http errors don't populate `headers` (at least apache digest auth challenge didn't).
osimons
parents:
645
diff
changeset
|
284 ).startswith('text/plain'): |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
628
diff
changeset
|
285 content = e.read() |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
628
diff
changeset
|
286 else: |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
287 content = 'no message available' |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
288 log.debug('Server returned error %d: %s (%s)', |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
289 e.code, e.msg, content) |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
290 raise |
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
291 return e |
29 | 292 |
392 | 293 def run(self): |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
294 if self.local: |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
295 fileobj = open(self.urls[0]) |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
296 try: |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
297 self._execute_build(None, fileobj) |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
298 finally: |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
299 fileobj.close() |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
300 return EX_OK |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
301 |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
302 urls = [] |
392 | 303 while True: |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
304 if not urls: |
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
305 urls[:] = self.urls |
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
306 url = urls.pop(0) |
392 | 307 try: |
308 try: | |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
309 if self.username and not self.auth_map.get(url): |
715
cd4405466a77
Add a `referer=` argument when logging in, and redirect to a Bitten URL so that slave doesn't need non-build permissions. Closes #459.
osimons
parents:
675
diff
changeset
|
310 login_url = '%s/login?referer=%s' % (url[:-7], |
cd4405466a77
Add a `referer=` argument when logging in, and redirect to a Bitten URL so that slave doesn't need non-build permissions. Closes #459.
osimons
parents:
675
diff
changeset
|
311 urllib.quote_plus(url)) |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
312 # First request to url, authentication needed |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
313 if self.form_auth: |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
314 log.debug('Performing http form authentication') |
715
cd4405466a77
Add a `referer=` argument when logging in, and redirect to a Bitten URL so that slave doesn't need non-build permissions. Closes #459.
osimons
parents:
675
diff
changeset
|
315 resp = self.request('POST', login_url) |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
316 match = FORM_TOKEN_RE.search(resp.read()) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
317 if not match: |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
318 log.error("Project %s does not support form " |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
319 "authentication" % url[:-7]) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
320 raise ExitSlave(EX_NOPERM) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
321 values = {'user': self.username, |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
322 'password': |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
323 self.password_mgr.find_user_password( |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
324 None, url)[1], |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
325 'referer': '', |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
326 '__FORM_TOKEN': match.group(1)} |
715
cd4405466a77
Add a `referer=` argument when logging in, and redirect to a Bitten URL so that slave doesn't need non-build permissions. Closes #459.
osimons
parents:
675
diff
changeset
|
327 self.request('POST', login_url, |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
328 body=urllib.urlencode(values)) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
329 else: |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
330 log.debug('Performing basic/digest authentication') |
715
cd4405466a77
Add a `referer=` argument when logging in, and redirect to a Bitten URL so that slave doesn't need non-build permissions. Closes #459.
osimons
parents:
675
diff
changeset
|
331 self.request('HEAD', login_url) |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
332 self.auth_map[url] = True |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
333 elif self.username: |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
334 log.debug('Reusing authentication information.') |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
335 else: |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
336 log.debug('Authentication not provided. Attempting to ' |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
337 'execute build anonymously.') |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
338 job_done = self._create_build(url) |
480
5181353ccb41
Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181
wbell
parents:
474
diff
changeset
|
339 if job_done: |
5181353ccb41
Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181
wbell
parents:
474
diff
changeset
|
340 continue |
465
04205ba5dc98
handle HTTPErrors separately to work around them not having a "reason" attribute like URLError
mgood
parents:
461
diff
changeset
|
341 except urllib2.HTTPError, e: |
04205ba5dc98
handle HTTPErrors separately to work around them not having a "reason" attribute like URLError
mgood
parents:
461
diff
changeset
|
342 # HTTPError doesn't have the "reason" attribute of URLError |
04205ba5dc98
handle HTTPErrors separately to work around them not having a "reason" attribute like URLError
mgood
parents:
461
diff
changeset
|
343 log.error(e) |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
344 raise ExitSlave(EX_UNAVAILABLE) |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
345 except urllib2.URLError, e: |
457
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
346 # Is this a temporary network glitch or something a bit |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
347 # more severe? |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
348 if isinstance(e.reason, socket.error) and \ |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
349 e.reason.args[0] in temp_net_errors: |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
350 log.warning(e) |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
351 else: |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
352 log.error(e) |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
353 raise ExitSlave(EX_UNAVAILABLE) |
490 | 354 except ExitSlave, e: |
355 return e.exit_code | |
525
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
356 if self.no_loop: |
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
357 break |
449
01503ed994e7
Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents:
442
diff
changeset
|
358 time.sleep(self.poll_interval) |
127
a9443f673344
Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29.
cmlenz
parents:
119
diff
changeset
|
359 |
392 | 360 def quit(self): |
361 log.info('Shutting down') | |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
362 raise ExitSlave(EX_OK) |
392 | 363 |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
364 def _create_build(self, url): |
649
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
648
diff
changeset
|
365 xml = xmlio.Element('slave', name=self.name, version=PROTOCOL_VERSION)[ |
233
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
366 xmlio.Element('platform', processor=self.config['processor'])[ |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
367 self.config['machine'] |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
368 ], |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
369 xmlio.Element('os', family=self.config['family'], |
244
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
242
diff
changeset
|
370 version=self.config['version'])[ |
233
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
371 self.config['os'] |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
372 ], |
29 | 373 ] |
392 | 374 |
375 log.debug('Configured packages: %s', self.config.packages) | |
233
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
376 for package, properties in self.config.packages.items(): |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
377 xml.append(xmlio.Element('package', name=package, **properties)) |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
378 |
392 | 379 body = str(xml) |
380 log.debug('Sending slave configuration: %s', body) | |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
381 resp = self.request('POST', url, body, { |
559
bf3e0c2f6b80
Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents:
549
diff
changeset
|
382 'Content-Length': str(len(body)), |
392 | 383 'Content-Type': 'application/x-bitten+xml' |
384 }) | |
277 | 385 |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
386 if resp.code == 201: |
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
387 self._initiate_build(resp.info().get('location')) |
480
5181353ccb41
Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181
wbell
parents:
474
diff
changeset
|
388 return True |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
389 elif resp.code == 204: |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
390 log.info('No pending builds') |
480
5181353ccb41
Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181
wbell
parents:
474
diff
changeset
|
391 return False |
392 | 392 else: |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
393 log.error('Unexpected response (%d %s)', resp.code, resp.msg) |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
394 raise ExitSlave(EX_PROTOCOL) |
56 | 395 |
392 | 396 def _initiate_build(self, build_url): |
418 | 397 log.info('Build pending at %s', build_url) |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
398 try: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
399 resp = self.request('GET', build_url) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
400 if resp.code == 200: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
401 self._execute_build(build_url, resp) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
402 else: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
403 log.error('Unexpected response (%d): %s', resp.code, resp.msg) |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
404 self._cancel_build(build_url, exit_code=EX_PROTOCOL) |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
405 except KeyboardInterrupt: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
406 log.warning('Build interrupted') |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
407 self._cancel_build(build_url) |
392 | 408 |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
409 def _execute_build(self, build_url, fileobj): |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
410 build_id = build_url and int(build_url.split('/')[-1]) or 0 |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
411 xml = xmlio.parse(fileobj) |
580
c66190747a9c
0.6dev: In the event that loading Recipe fails, `basedir` would not yet have been defined, and removing it would fail.
osimons
parents:
576
diff
changeset
|
412 basedir = '' |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
413 try: |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
414 if not self.local: |
796
031ce03d2baf
Don't initialize the keep-alive thread when running a local build (it will fail trying to get the opener).
osimons
parents:
785
diff
changeset
|
415 keepalive_thread = KeepAliveThread(self.opener, build_url, |
031ce03d2baf
Don't initialize the keep-alive thread when running a local build (it will fail trying to get the opener).
osimons
parents:
785
diff
changeset
|
416 self.single_build, self.keepalive_interval) |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
417 keepalive_thread.start() |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
418 recipe = Recipe(xml, os.path.join(self.work_dir, self.build_dir), |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
419 self.config) |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
420 basedir = recipe.ctxt.basedir |
490 | 421 log.debug('Running build in directory %s' % basedir) |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
422 if not os.path.exists(basedir): |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
423 os.mkdir(basedir) |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
424 |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
425 for step in recipe: |
754
545be0c8f405
Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents:
719
diff
changeset
|
426 try: |
545be0c8f405
Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents:
719
diff
changeset
|
427 log.info('Executing build step %r, onerror = %s', step.id, step.onerror) |
545be0c8f405
Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents:
719
diff
changeset
|
428 if not self._execute_step(build_url, recipe, step): |
545be0c8f405
Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents:
719
diff
changeset
|
429 log.warning('Stopping build due to failure') |
545be0c8f405
Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents:
719
diff
changeset
|
430 break |
545be0c8f405
Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents:
719
diff
changeset
|
431 except Exception, e: |
545be0c8f405
Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents:
719
diff
changeset
|
432 log.error('Exception raised processing step %s. Reraising %s', step.id, e) |
545be0c8f405
Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents:
719
diff
changeset
|
433 raise |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
434 else: |
417
ab2557ff0d97
Change the default log verbosity of the build slave, and remove the `--debug` option.
cmlenz
parents:
414
diff
changeset
|
435 log.info('Build completed') |
461
f487963dd886
When the slave is run in `--dry-run` mode, it now also sends a build cancellation message to the master in case of failed builds. Closes #187.
cmlenz
parents:
457
diff
changeset
|
436 if self.dry_run: |
f487963dd886
When the slave is run in `--dry-run` mode, it now also sends a build cancellation message to the master in case of failed builds. Closes #187.
cmlenz
parents:
457
diff
changeset
|
437 self._cancel_build(build_url) |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
438 finally: |
796
031ce03d2baf
Don't initialize the keep-alive thread when running a local build (it will fail trying to get the opener).
osimons
parents:
785
diff
changeset
|
439 if not self.local: |
031ce03d2baf
Don't initialize the keep-alive thread when running a local build (it will fail trying to get the opener).
osimons
parents:
785
diff
changeset
|
440 keepalive_thread.stop() |
580
c66190747a9c
0.6dev: In the event that loading Recipe fails, `basedir` would not yet have been defined, and removing it would fail.
osimons
parents:
576
diff
changeset
|
441 if not self.keep_files and os.path.isdir(basedir): |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
442 log.debug('Removing build directory %s' % basedir) |
474
8c3dfe0efe78
Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents:
466
diff
changeset
|
443 _rmtree(basedir) |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
444 if self.single_build: |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
445 log.info('Exiting after single build completed.') |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
446 raise ExitSlave(EX_OK) |
392 | 447 |
448 def _execute_step(self, build_url, recipe, step): | |
449 failed = False | |
758
d11ef8024d7c
Make all times be generated by the server. This eliminates a number of inconsistencies you'll see when the clocks on the slaves are skewed from the master.
wbell
parents:
754
diff
changeset
|
450 started = int(time.time()) |
d11ef8024d7c
Make all times be generated by the server. This eliminates a number of inconsistencies you'll see when the clocks on the slaves are skewed from the master.
wbell
parents:
754
diff
changeset
|
451 xml = xmlio.Element('result', step=step.id) |
392 | 452 try: |
453 for type, category, generator, output in \ | |
454 step.execute(recipe.ctxt): | |
455 if type == Recipe.ERROR: | |
82
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
456 failed = True |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
457 if type == Recipe.REPORT and self.dump_reports: |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
458 print output |
835
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
459 if type == Recipe.ATTACH: |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
460 # Attachments are added out-of-band due to major |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
461 # performance issues with inlined base64 xml content |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
462 self._attach_file(build_url, recipe, output) |
392 | 463 xml.append(xmlio.Element(type, category=category, |
464 generator=generator)[ | |
465 output | |
466 ]) | |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
467 except KeyboardInterrupt: |
417
ab2557ff0d97
Change the default log verbosity of the build slave, and remove the `--debug` option.
cmlenz
parents:
414
diff
changeset
|
468 log.warning('Build interrupted') |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
469 self._cancel_build(build_url) |
392 | 470 except BuildError, e: |
628
05686657e989
0.6dev: Implement a `subprocess` implementation of `execute()` that is now the default for all slaves where this is available - essentially all slaves with Python 2.4 and higher, or where module is installed separately. This fixes:
osimons
parents:
610
diff
changeset
|
471 log.error('Build step %r failed', step.id) |
392 | 472 failed = True |
473 except Exception, e: | |
474 log.error('Internal error in build step %r', step.id, exc_info=True) | |
475 failed = True | |
758
d11ef8024d7c
Make all times be generated by the server. This eliminates a number of inconsistencies you'll see when the clocks on the slaves are skewed from the master.
wbell
parents:
754
diff
changeset
|
476 xml.attr['duration'] = (time.time() - started) |
392 | 477 if failed: |
478 xml.attr['status'] = 'failure' | |
479 else: | |
480 xml.attr['status'] = 'success' | |
481 log.info('Build step %s completed successfully', step.id) | |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
482 |
424
bc81d7b069ff
Rudimentary implementation of ``dry-run`` option in build slave. Builds are still put in in progress mode, but no results are transmitted back, and the build is automatically invalidated after the slave is done.
cmlenz
parents:
420
diff
changeset
|
483 if not self.local and not self.dry_run: |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
484 try: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
485 resp = self.request('POST', build_url + '/steps/', str(xml), { |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
486 'Content-Type': 'application/x-bitten+xml' |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
487 }) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
488 if resp.code != 201: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
489 log.error('Unexpected response (%d): %s', resp.code, |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
490 resp.msg) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
491 except KeyboardInterrupt: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
492 log.warning('Build interrupted') |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
493 self._cancel_build(build_url) |
392 | 494 return not failed or step.onerror != 'fail' |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
495 |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
496 def _cancel_build(self, build_url, exit_code=EX_OK): |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
497 log.info('Cancelling build at %s', build_url) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
498 if not self.local: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
499 resp = self.request('DELETE', build_url) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
500 if resp.code not in (200, 204): |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
501 log.error('Unexpected response (%d): %s', resp.code, resp.msg) |
490 | 502 raise ExitSlave(exit_code) |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
503 |
835
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
504 def _attach_file(self, build_url, recipe, attachment): |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
505 form_token = recipe._root.attr.get('form_token', '') |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
506 if self.local or self.dry_run or not form_token: |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
507 log.info('Attachment %s not sent due to current slave options', |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
508 attachment.attr['file']) |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
509 return |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
510 resource_type = attachment.attr['resource'] |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
511 url = str(build_url + '/attach/' + resource_type) |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
512 path = recipe.ctxt.resolve(attachment.attr['filename']) |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
513 filename = os.path.basename(path) |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
514 log.debug('Attaching file %s to %s...', attachment.attr['filename'], |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
515 resource_type) |
889
9ac91d38198c
Make sure slave reads attachments as binary files. Fixes #657.
osimons
parents:
870
diff
changeset
|
516 f = open(path, 'rb') |
835
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
517 try: |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
518 data, content_type = encode_multipart_formdata({ |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
519 'file': (filename, f.read()), |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
520 'description': attachment.attr['description'], |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
521 '__FORM_TOKEN': form_token}) |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
522 finally: |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
523 f.close() |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
524 resp = self.request('POST', url , data, { |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
525 'Content-Type': content_type}) |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
526 if not resp.code == 201: |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
527 msg = 'Error attaching %s to %s' |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
528 log.error(msg, filename, resource_type) |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
529 raise BuildError(msg, filename, resource_type) |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
530 |
392 | 531 class ExitSlave(Exception): |
532 """Exception used internally by the slave to signal that the slave process | |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
533 should be stopped. |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
534 """ |
490 | 535 def __init__(self, exit_code): |
536 self.exit_code = exit_code | |
537 Exception.__init__(self) | |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
538 |
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
539 |
31 | 540 def main(): |
313 | 541 """Main entry point for running the build slave.""" |
56 | 542 from bitten import __version__ as VERSION |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
543 from optparse import OptionParser |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
544 |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
545 parser = OptionParser(usage='usage: %prog [options] url1 [url2] ...', |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
546 version='%%prog %s' % VERSION) |
185
2c24d9a950ed
Add a `--dry-run` option to the build slave. This will result in the slave being registered and executing builds, but without submitting the progress and results of the build back to the server. Useful for getting the configuration of new slaves right without polluting the database with invalid builds.
cmlenz
parents:
163
diff
changeset
|
547 parser.add_option('--name', action='store', dest='name', |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
548 help='name of this slave (defaults to host name)') |
208
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
549 parser.add_option('-f', '--config', action='store', dest='config', |
127
a9443f673344
Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29.
cmlenz
parents:
119
diff
changeset
|
550 metavar='FILE', help='path to configuration file') |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
551 parser.add_option('-u', '--user', dest='username', |
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
552 help='the username to use for authentication') |
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
553 parser.add_option('-p', '--password', dest='password', |
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
554 help='the password to use when authenticating') |
587
0ac21e3343a4
0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents:
580
diff
changeset
|
555 def _ask_password(option, opt_str, value, parser): |
0ac21e3343a4
0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents:
580
diff
changeset
|
556 from getpass import getpass |
0ac21e3343a4
0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents:
580
diff
changeset
|
557 parser.values.password = getpass('Passsword: ') |
0ac21e3343a4
0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents:
580
diff
changeset
|
558 parser.add_option('-P', '--ask-password', action='callback', |
0ac21e3343a4
0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents:
580
diff
changeset
|
559 callback=_ask_password, help='Prompt for password') |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
560 parser.add_option('--form-auth', action='store_true', |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
561 dest='form_auth', |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
562 help='login using AccountManager HTML form instead of ' |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
563 'HTTP authentication for all urls') |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
564 |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
565 group = parser.add_option_group('building') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
566 group.add_option('-d', '--work-dir', action='store', dest='work_dir', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
567 metavar='DIR', help='working directory for builds') |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
568 group.add_option('--build-dir', action='store', dest='build_dir', |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
569 default = 'build_${config}_${build}', |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
570 help='name pattern for the build dir to use inside the ' |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
571 'working dir ["%default"]') |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
572 group.add_option('-k', '--keep-files', action='store_true', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
573 dest='keep_files', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
574 help='don\'t delete files after builds') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
575 group.add_option('-s', '--single', action='store_true', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
576 dest='single_build', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
577 help='exit after completing a single build') |
525
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
578 group.add_option('', '--no-loop', action='store_true', |
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
579 dest='no_loop', |
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
580 help='exit after completing a single check and running ' |
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
581 'the required builds') |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
582 group.add_option('-n', '--dry-run', action='store_true', dest='dry_run', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
583 help='don\'t report results back to master') |
449
01503ed994e7
Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents:
442
diff
changeset
|
584 group.add_option('-i', '--interval', dest='interval', metavar='SECONDS', |
01503ed994e7
Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents:
442
diff
changeset
|
585 type='int', help='time to wait between requesting builds') |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
586 group.add_option('-b', '--keepalive_interval', dest='keepalive_interval', metavar='SECONDS', type='int', help='time to wait between keepalive heartbeats') |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
587 group = parser.add_option_group('logging') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
588 group.add_option('-l', '--log', dest='logfile', metavar='FILENAME', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
589 help='write log messages to FILENAME') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
590 group.add_option('-v', '--verbose', action='store_const', dest='loglevel', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
591 const=logging.DEBUG, help='print as much as possible') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
592 group.add_option('-q', '--quiet', action='store_const', dest='loglevel', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
593 const=logging.WARN, help='print as little as possible') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
594 group.add_option('--dump-reports', action='store_true', dest='dump_reports', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
595 help='whether report data should be printed') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
596 |
242
372d1de2e3ec
* Fixes to the `<c:configure>` command added in [247]: Set current directory when invoking the script, and correctly pass `CFLAGS` and `CXXFLAGS`.
cmlenz
parents:
241
diff
changeset
|
597 parser.set_defaults(dry_run=False, keep_files=False, |
525
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
598 loglevel=logging.INFO, single_build=False, no_loop=False, |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
599 dump_reports=False, interval=300, keepalive_interval=60, |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
600 form_auth=False) |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
601 options, args = parser.parse_args() |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
602 |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
603 if len(args) < 1: |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
604 parser.error('incorrect number of arguments') |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
605 urls = args |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
606 |
157
2efdc69e63c3
Some style/documentation improvements to make Pylint happier.
cmlenz
parents:
149
diff
changeset
|
607 logger = logging.getLogger('bitten') |
2efdc69e63c3
Some style/documentation improvements to make Pylint happier.
cmlenz
parents:
149
diff
changeset
|
608 logger.setLevel(options.loglevel) |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
90
diff
changeset
|
609 handler = logging.StreamHandler() |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
90
diff
changeset
|
610 handler.setLevel(options.loglevel) |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
90
diff
changeset
|
611 formatter = logging.Formatter('[%(levelname)-8s] %(message)s') |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
90
diff
changeset
|
612 handler.setFormatter(formatter) |
157
2efdc69e63c3
Some style/documentation improvements to make Pylint happier.
cmlenz
parents:
149
diff
changeset
|
613 logger.addHandler(handler) |
339
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
614 if options.logfile: |
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
615 handler = logging.FileHandler(options.logfile) |
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
616 handler.setLevel(options.loglevel) |
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
617 formatter = logging.Formatter('%(asctime)s [%(name)s] %(levelname)s: ' |
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
618 '%(message)s') |
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
619 handler.setFormatter(formatter) |
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
620 logger.addHandler(handler) |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
621 |
610
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
622 log.info("Slave launched at %s" % \ |
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
623 datetime.now().strftime('%Y-%m-%d %H:%M:%S')) |
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
624 |
675
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
625 slave = None |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
626 try: |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
627 slave = BuildSlave(urls, name=options.name, config=options.config, |
392 | 628 dry_run=options.dry_run, work_dir=options.work_dir, |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
629 build_dir=options.build_dir, |
392 | 630 keep_files=options.keep_files, |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
631 single_build=options.single_build, |
543
ce16a5dcf8b3
Pass `--no-loop` option to the `BuildSlave` (fixes #375, from the patch there - thanks!)
dfraser
parents:
542
diff
changeset
|
632 no_loop=options.no_loop, |
449
01503ed994e7
Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents:
442
diff
changeset
|
633 poll_interval=options.interval, |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
634 keepalive_interval=options.keepalive_interval, |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
635 username=options.username, password=options.password, |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
636 dump_reports=options.dump_reports, |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
637 form_auth=options.form_auth) |
392 | 638 try: |
490 | 639 exit_code = slave.run() |
392 | 640 except KeyboardInterrupt: |
641 slave.quit() | |
675
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
642 except ConfigFileNotFound, e: |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
643 log.error(e) |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
644 exit_code = EX_IOERR |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
645 except MissingSectionHeaderError: |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
646 log.error("Error parsing configuration file %r. Wrong format?" \ |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
647 % options.config) |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
648 exit_code = EX_IOERR |
490 | 649 except ExitSlave, e: |
650 exit_code = e.exit_code | |
31 | 651 |
675
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
652 if slave and not (options.work_dir or options.keep_files): |
576
4b75e3f6f27c
0.6dev: Fixing #229 again - keeping `slave.work_dir` if either `--keep_files` or `--work_dir=` are part of options.
osimons
parents:
570
diff
changeset
|
653 log.debug('Removing working directory %s' % slave.work_dir) |
474
8c3dfe0efe78
Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents:
466
diff
changeset
|
654 _rmtree(slave.work_dir) |
610
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
655 |
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
656 log.info("Slave exited at %s" % \ |
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
657 datetime.now().strftime('%Y-%m-%d %H:%M:%S')) |
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
658 |
490 | 659 return exit_code |
208
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
660 |
31 | 661 if __name__ == '__main__': |
490 | 662 sys.exit(main()) |