annotate bitten/slave.py @ 895:7d93d6358fe0

Use our own HTTPBasicAuthHandler under Python 2.6 to avoid issue http://bugs.python.org/issue8797. Fixes #658.
author hodgestar
date Wed, 09 Mar 2011 14:48:35 +0000
parents 60d0807334ec
children ce35ae342a66
rev   line source
379
0df178e07fdb Use UTF-8 as encoding of source files.
cmlenz
parents: 377
diff changeset
1 # -*- coding: utf-8 -*-
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
2 #
832
7c80375d4817 Updated copyright to 2010.
osimons
parents: 796
diff changeset
3 # Copyright (C) 2007-2010 Edgewall Software
408
933105ab516b Update file headers and other stuff pointing to the old home.
cmlenz
parents: 405
diff changeset
4 # Copyright (C) 2005-2007 Christopher Lenz <cmlenz@gmx.de>
163
634be6cbb808 Flip the switch: Bitten is now BSD-licensed.
cmlenz
parents: 159
diff changeset
5 # All rights reserved.
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
6 #
163
634be6cbb808 Flip the switch: Bitten is now BSD-licensed.
cmlenz
parents: 159
diff changeset
7 # This software is licensed as described in the file COPYING, which
634be6cbb808 Flip the switch: Bitten is now BSD-licensed.
cmlenz
parents: 159
diff changeset
8 # you should have received as part of this distribution. The terms
408
933105ab516b Update file headers and other stuff pointing to the old home.
cmlenz
parents: 405
diff changeset
9 # are also available at http://bitten.edgewall.org/wiki/License.
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
10
313
90422699a594 More and improved docstrings (using epydoc format).
cmlenz
parents: 310
diff changeset
11 """Implementation of the build slave."""
90422699a594 More and improved docstrings (using epydoc format).
cmlenz
parents: 310
diff changeset
12
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
13 from datetime import datetime
457
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and
jonas
parents: 456
diff changeset
14 import errno
648
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
15 import urllib
402
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents: 401
diff changeset
16 import urllib2
15
06207499c58c * Use logging in the BEEP core as well as in the master and slave scripts. Closes #4.
cmlenz
parents: 14
diff changeset
17 import logging
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
18 import os
66
df40a4eb7553 Use the standard {{{platform}}} module for reporting of system info by slaves. Closes #6.
cmlenz
parents: 63
diff changeset
19 import platform
90
2c4e104afef8 The build master now transmits snapshot archives without blocking while reading the file and sending the BEEP frames. Closes #17.
cmlenz
parents: 88
diff changeset
20 import shutil
457
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and
jonas
parents: 456
diff changeset
21 import socket
654
cc1fefd5d814 0.6dev: Moved a constant declaration so that master should still work with Python 2.4. Updated docs with regards to Python requirements + added a changelog message from cmlenz.
osimons
parents: 652
diff changeset
22 import sys
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
23 import tempfile
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
24 import time
648
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
25 import re
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
26 import cookielib
785
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
27 import threading
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
28 import os
835
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
29 import mimetools
675
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents: 654
diff changeset
30 from ConfigParser import MissingSectionHeaderError
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
31
654
cc1fefd5d814 0.6dev: Moved a constant declaration so that master should still work with Python 2.4. Updated docs with regards to Python requirements + added a changelog message from cmlenz.
osimons
parents: 652
diff changeset
32 from bitten import PROTOCOL_VERSION
61
47ab019508dd Moved {{{BuildError}}} class into package {{{bitten.build}}}.
cmlenz
parents: 60
diff changeset
33 from bitten.build import BuildError
675
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents: 654
diff changeset
34 from bitten.build.config import Configuration, ConfigFileNotFound
436
cfbc9ee622d5 Finish the move of build configuration management into the admin interface.
cmlenz
parents: 424
diff changeset
35 from bitten.recipe import Recipe
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
36 from bitten.util import xmlio
895
7d93d6358fe0 Use our own HTTPBasicAuthHandler under Python 2.6 to avoid issue http://bugs.python.org/issue8797. Fixes #658.
hodgestar
parents: 891
diff changeset
37 from bitten.util.compat import HTTPBasicAuthHandler
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
38
495
7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents: 494
diff changeset
39 EX_OK = getattr(os, "EX_OK", 0)
7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents: 494
diff changeset
40 EX_UNAVAILABLE = getattr(os, "EX_UNAVAILABLE", 69)
675
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents: 654
diff changeset
41 EX_IOERR = getattr(os, "EX_IOERR", 74)
495
7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents: 494
diff changeset
42 EX_PROTOCOL = getattr(os, "EX_PROTOCOL", 76)
648
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
43 EX_NOPERM = getattr(os, "EX_NOPERM", 77)
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
44
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
45 FORM_TOKEN_RE = re.compile('__FORM_TOKEN\" value=\"(.+)\"')
495
7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents: 494
diff changeset
46
411
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
47 __all__ = ['BuildSlave', 'ExitSlave']
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
48 __docformat__ = 'restructuredtext en'
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
49
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 90
diff changeset
50 log = logging.getLogger('bitten.slave')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 90
diff changeset
51
457
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and
jonas
parents: 456
diff changeset
52 # List of network errors which are usually temporary and non critical.
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and
jonas
parents: 456
diff changeset
53 temp_net_errors = [errno.ENETUNREACH, errno.ENETDOWN, errno.ETIMEDOUT,
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and
jonas
parents: 456
diff changeset
54 errno.ECONNREFUSED]
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and
jonas
parents: 456
diff changeset
55
474
8c3dfe0efe78 Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents: 466
diff changeset
56 def _rmtree(root):
549
cb137f2121b9 Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents: 545
diff changeset
57 """Catch shutil.rmtree failures on Windows when files are read-only, and only remove if root exists."""
474
8c3dfe0efe78 Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents: 466
diff changeset
58 def _handle_error(fn, path, excinfo):
8c3dfe0efe78 Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents: 466
diff changeset
59 os.chmod(path, 0666)
8c3dfe0efe78 Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents: 466
diff changeset
60 fn(path)
549
cb137f2121b9 Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents: 545
diff changeset
61 if os.path.exists(root):
cb137f2121b9 Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents: 545
diff changeset
62 return shutil.rmtree(root, onerror=_handle_error)
cb137f2121b9 Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents: 545
diff changeset
63 else:
cb137f2121b9 Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents: 545
diff changeset
64 return False
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
65
402
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents: 401
diff changeset
66
420
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
67 class SaneHTTPRequest(urllib2.Request):
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
68
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
69 def __init__(self, method, url, data=None, headers={}):
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
70 urllib2.Request.__init__(self, url, data, headers)
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
71 self.method = method
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
72
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
73 def get_method(self):
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
74 if self.method is None:
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
75 self.method = self.has_data() and 'POST' or 'GET'
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
76 return self.method
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
77
835
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
78
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
79 def encode_multipart_formdata(fields):
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
80 """
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
81 Given a dictionary field parameters, returns the HTTP request body and the
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
82 content_type (which includes the boundary string), to be used with an
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
83 httplib-like call.
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
84
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
85 Normal key/value items are treated as regular parameters, but key/tuple
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
86 items are treated as files, where a value tuple is a (filename, data) tuple.
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
87
870
c837eaf8ff2e Fix docstring complaints from epydoc.
hodgestar
parents: 835
diff changeset
88 For example::
c837eaf8ff2e Fix docstring complaints from epydoc.
hodgestar
parents: 835
diff changeset
89
835
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
90 fields = {
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
91 'foo': 'bar',
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
92 'foofile': ('foofile.txt', 'contents of foofile'),
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
93 }
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
94 body, content_type = encode_multipart_formdata(fields)
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
95
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
96 Note: Adapted from http://code.google.com/p/urllib3/ (MIT license)
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
97 """
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
98
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
99 BOUNDARY = mimetools.choose_boundary()
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
100 ENCODE_TEMPLATE= "--%(boundary)s\r\n" \
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
101 "Content-Disposition: form-data; name=\"%(name)s\"\r\n" \
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
102 "\r\n%(value)s\r\n"
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
103 ENCODE_TEMPLATE_FILE = "--%(boundary)s\r\n" \
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
104 "Content-Disposition: form-data; name=\"%(name)s\"; " \
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
105 "filename=\"%(filename)s\"\r\n" \
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
106 "Content-Type: %(contenttype)s\r\n" \
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
107 "\r\n%(value)s\r\n"
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
108
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
109 body = ""
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
110 for key, value in fields.iteritems():
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
111 if isinstance(value, tuple):
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
112 filename, value = value
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
113 body += ENCODE_TEMPLATE_FILE % {
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
114 'boundary': BOUNDARY,
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
115 'name': str(key),
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
116 'value': str(value),
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
117 'filename': str(filename),
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
118 'contenttype': 'application/octet-stream'
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
119 }
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
120 else:
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
121 body += ENCODE_TEMPLATE % {
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
122 'boundary': BOUNDARY,
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
123 'name': str(key),
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
124 'value': str(value)
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
125 }
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
126 body += '--%s--\r\n' % BOUNDARY
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
127 content_type = 'multipart/form-data; boundary=%s' % BOUNDARY
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
128 return body, content_type
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
129
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
130
785
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
131 class KeepAliveThread(threading.Thread):
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
132 "A thread to periodically send keep-alive messages to the master"
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
133
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
134 def __init__(self, opener, build_url, single_build, keepalive_interval):
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
135 threading.Thread.__init__(self, None, None, "KeepaliveThread")
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
136 self.build_url = build_url
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
137 self.keepalive_interval = keepalive_interval
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
138 self.single_build = single_build
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
139 self.last_keepalive = int(time.time())
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
140 self.kill = False
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
141 self.opener = opener
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
142
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
143 def keepalive(self):
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
144 log.debug('Sending keepalive')
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
145 method = 'POST'
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
146 url = self.build_url + '/keepalive/'
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
147 body = None
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
148 shutdown = False
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
149 headers = {
891
60d0807334ec Set 'Content-Length' header on keep-alive requests. Fixes #667. Thanks eblot.
osimons
parents: 889
diff changeset
150 'Content-Type': 'application/x-bitten+xml',
60d0807334ec Set 'Content-Length' header on keep-alive requests. Fixes #667. Thanks eblot.
osimons
parents: 889
diff changeset
151 'Content-Length': '0'
785
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
152 }
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
153 log.debug('Sending %s request to %r', method, url)
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
154 req = SaneHTTPRequest(method, url, body, headers or {})
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
155 try:
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
156 return self.opener.open(req)
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
157 except urllib2.HTTPError, e:
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
158 # a conflict error lets us know that we've been
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
159 # invalidated. Ideally, we'd engineer something to stop any
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
160 # running steps in progress, but killing threads is tricky
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
161 # stuff. For now, we'll wait for whatever's going
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
162 # on to stop, and the main thread'll figure out that we've
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
163 # been invalidated.
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
164 log.warning('Server returned keepalive error %d: %s', e.code, e.msg)
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
165 except:
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
166 log.warning('Server returned unknown keepalive error')
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
167
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
168 def run(self):
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
169 log.debug('Keepalive thread starting.')
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
170 while (not self.kill):
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
171 now = int(time.time())
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
172 if (self.last_keepalive + self.keepalive_interval) < now:
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
173 self.keepalive()
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
174 self.last_keepalive = now
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
175
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
176 time.sleep(1)
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
177 log.debug('Keepalive thread exiting.')
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
178
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
179 def stop(self):
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
180 log.debug('Stopping keepalive thread')
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
181 self.kill = True
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
182 self.join(30)
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
183 log.debug('Keepalive thread stopped')
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
184
420
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
185
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
186 class BuildSlave(object):
542
cacf018eb190 Small documentation fixes.
wbell
parents: 525
diff changeset
187 """HTTP client implementation for the build slave."""
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
188
494
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents: 490
diff changeset
189 def __init__(self, urls, name=None, config=None, dry_run=False,
466
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents: 465
diff changeset
190 work_dir=None, build_dir="build_${build}",
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents: 465
diff changeset
191 keep_files=False, single_build=False,
785
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
192 poll_interval=300, keepalive_interval = 60,
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
193 username=None, password=None,
648
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
194 dump_reports=False, no_loop=False, form_auth=False):
313
90422699a594 More and improved docstrings (using epydoc format).
cmlenz
parents: 310
diff changeset
195 """Create the build slave instance.
90422699a594 More and improved docstrings (using epydoc format).
cmlenz
parents: 310
diff changeset
196
494
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents: 490
diff changeset
197 :param urls: a list of URLs of the build masters to connect to, or a
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents: 490
diff changeset
198 single-element list containing the path to a build recipe
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents: 490
diff changeset
199 file
414
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
200 :param name: the name with which this slave should identify itself
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
201 :param config: the path to the slave configuration file
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
202 :param dry_run: wether the build outcome should not be reported back
411
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
203 to the master
414
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
204 :param work_dir: the working directory to use for build execution
466
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents: 465
diff changeset
205 :param build_dir: the pattern to use for naming the build subdir
414
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
206 :param keep_files: whether files and directories created for build
411
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
207 execution should be kept when done
414
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
208 :param single_build: whether this slave should exit after completing a
411
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
209 single build, or continue processing builds forever
449
01503ed994e7 Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents: 442
diff changeset
210 :param poll_interval: the time in seconds to wait between requesting
451
57822fefd441 Fix default for poll interval of slave.
cmlenz
parents: 449
diff changeset
211 builds from the build master (default is five
57822fefd441 Fix default for poll interval of slave.
cmlenz
parents: 449
diff changeset
212 minutes)
870
c837eaf8ff2e Fix docstring complaints from epydoc.
hodgestar
parents: 835
diff changeset
213 :param keepalive_interval: the time in seconds to wait between sending
785
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
214 keepalive heartbeats (default is 30 seconds)
411
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
215 :param username: the username to use when authentication against the
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
216 build master is requested
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
217 :param password: the password to use when authentication is needed
442
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
218 :param dump_reports: whether report data should be written to the
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
219 standard output, in addition to being transmitted
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
220 to the build master
525
bb448ad9bf35 Added option for not looping after check. Fixes #371
dfraser
parents: 515
diff changeset
221 :param no_loop: for this slave to just perform a single check, regardless
bb448ad9bf35 Added option for not looping after check. Fixes #371
dfraser
parents: 515
diff changeset
222 of whether a build is done or not
648
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
223 :param form_auth: login using AccountManager HTML form instead of
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
224 HTTP authentication for all urls
313
90422699a594 More and improved docstrings (using epydoc format).
cmlenz
parents: 310
diff changeset
225 """
494
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents: 490
diff changeset
226 self.urls = urls
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents: 490
diff changeset
227 self.local = len(urls) == 1 and not urls[0].startswith('http://') \
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents: 490
diff changeset
228 and not urls[0].startswith('https://')
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
229 if name is None:
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
230 name = platform.node().split('.', 1)[0].lower()
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
231 self.name = name
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
232 self.config = Configuration(config)
185
2c24d9a950ed Add a `--dry-run` option to the build slave. This will result in the slave being registered and executing builds, but without submitting the progress and results of the build back to the server. Useful for getting the configuration of new slaves right without polluting the database with invalid builds.
cmlenz
parents: 163
diff changeset
233 self.dry_run = dry_run
208
f39136272683 * Slave now removes build working directories when done.
cmlenz
parents: 203
diff changeset
234 if not work_dir:
f39136272683 * Slave now removes build working directories when done.
cmlenz
parents: 203
diff changeset
235 work_dir = tempfile.mkdtemp(prefix='bitten')
f39136272683 * Slave now removes build working directories when done.
cmlenz
parents: 203
diff changeset
236 elif not os.path.exists(work_dir):
f39136272683 * Slave now removes build working directories when done.
cmlenz
parents: 203
diff changeset
237 os.makedirs(work_dir)
f39136272683 * Slave now removes build working directories when done.
cmlenz
parents: 203
diff changeset
238 self.work_dir = work_dir
466
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents: 465
diff changeset
239 self.build_dir = build_dir
241
d3c8a74922cd * Add an option to `bitten-slave` that tells it to keep any files in the working directory.
cmlenz
parents: 233
diff changeset
240 self.keep_files = keep_files
377
a465b5dbfecf Tabs and line length.
cmlenz
parents: 373
diff changeset
241 self.single_build = single_build
525
bb448ad9bf35 Added option for not looping after check. Fixes #371
dfraser
parents: 515
diff changeset
242 self.no_loop = no_loop
449
01503ed994e7 Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents: 442
diff changeset
243 self.poll_interval = poll_interval
785
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
244 self.keepalive_interval = keepalive_interval
442
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
245 self.dump_reports = dump_reports
648
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
246 self.cookiejar = cookielib.CookieJar()
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
247 self.username = username \
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
248 or self.config['authentication.username'] or ''
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
249
414
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
250 if not self.local:
569
74bc339ce7d5 0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents: 560
diff changeset
251 self.password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
648
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
252 if self.username:
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
253 log.debug('Enabling authentication with username %r',
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
254 self.username)
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
255 self.form_auth = form_auth
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
256 password = password \
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
257 or self.config['authentication.password'] or ''
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
258 self.config.packages.pop('authentication', None)
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
259 urls = [url[:-7] for url in urls]
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
260 self.password_mgr.add_password(
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
261 None, urls, self.username, password)
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
262 self.auth_map = dict(map(lambda x: (x, False), urls))
569
74bc339ce7d5 0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents: 560
diff changeset
263
74bc339ce7d5 0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents: 560
diff changeset
264 def _get_opener(self):
652
de04ce69da53 0.6dev: Removing code and updated docs related to Trac < 0.11 and Python < 2.4 (base requirements for Bitten 0.6).
osimons
parents: 649
diff changeset
265 opener = urllib2.build_opener(urllib2.HTTPErrorProcessor())
895
7d93d6358fe0 Use our own HTTPBasicAuthHandler under Python 2.6 to avoid issue http://bugs.python.org/issue8797. Fixes #658.
hodgestar
parents: 891
diff changeset
266 opener.add_handler(HTTPBasicAuthHandler(self.password_mgr))
569
74bc339ce7d5 0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents: 560
diff changeset
267 opener.add_handler(urllib2.HTTPDigestAuthHandler(self.password_mgr))
648
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
268 opener.add_handler(urllib2.HTTPCookieProcessor(self.cookiejar))
569
74bc339ce7d5 0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents: 560
diff changeset
269 return opener
74bc339ce7d5 0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents: 560
diff changeset
270 opener = property(_get_opener)
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
271
402
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents: 401
diff changeset
272 def request(self, method, url, body=None, headers=None):
420
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
273 log.debug('Sending %s request to %r', method, url)
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
274 req = SaneHTTPRequest(method, url, body, headers or {})
402
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents: 401
diff changeset
275 try:
559
bf3e0c2f6b80 Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents: 549
diff changeset
276 resp = self.opener.open(req)
bf3e0c2f6b80 Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents: 549
diff changeset
277 if not hasattr(resp, 'code'):
bf3e0c2f6b80 Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents: 549
diff changeset
278 resp.code = 200
bf3e0c2f6b80 Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents: 549
diff changeset
279 return resp
402
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents: 401
diff changeset
280 except urllib2.HTTPError, e:
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents: 401
diff changeset
281 if e.code >= 300:
647
6fe6fc8ee36c 0.6dev: Follow-up to [719]. Seems some http errors don't populate `headers` (at least apache digest auth challenge didn't).
osimons
parents: 645
diff changeset
282 if hasattr(e, 'headers') and \
6fe6fc8ee36c 0.6dev: Follow-up to [719]. Seems some http errors don't populate `headers` (at least apache digest auth challenge didn't).
osimons
parents: 645
diff changeset
283 e.headers.getheader('Content-Type', ''
6fe6fc8ee36c 0.6dev: Follow-up to [719]. Seems some http errors don't populate `headers` (at least apache digest auth challenge didn't).
osimons
parents: 645
diff changeset
284 ).startswith('text/plain'):
645
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents: 628
diff changeset
285 content = e.read()
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents: 628
diff changeset
286 else:
785
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
287 content = 'no message available'
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
288 log.debug('Server returned error %d: %s (%s)',
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
289 e.code, e.msg, content)
402
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents: 401
diff changeset
290 raise
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents: 401
diff changeset
291 return e
29
2adc3480e4aa Some cleanup and additional docstrings.
cmlenz
parents: 19
diff changeset
292
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
293 def run(self):
414
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
294 if self.local:
494
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents: 490
diff changeset
295 fileobj = open(self.urls[0])
414
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
296 try:
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
297 self._execute_build(None, fileobj)
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
298 finally:
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
299 fileobj.close()
495
7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents: 494
diff changeset
300 return EX_OK
414
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
301
494
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents: 490
diff changeset
302 urls = []
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
303 while True:
494
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents: 490
diff changeset
304 if not urls:
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents: 490
diff changeset
305 urls[:] = self.urls
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents: 490
diff changeset
306 url = urls.pop(0)
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
307 try:
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
308 try:
648
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
309 if self.username and not self.auth_map.get(url):
715
cd4405466a77 Add a `referer=` argument when logging in, and redirect to a Bitten URL so that slave doesn't need non-build permissions. Closes #459.
osimons
parents: 675
diff changeset
310 login_url = '%s/login?referer=%s' % (url[:-7],
cd4405466a77 Add a `referer=` argument when logging in, and redirect to a Bitten URL so that slave doesn't need non-build permissions. Closes #459.
osimons
parents: 675
diff changeset
311 urllib.quote_plus(url))
648
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
312 # First request to url, authentication needed
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
313 if self.form_auth:
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
314 log.debug('Performing http form authentication')
715
cd4405466a77 Add a `referer=` argument when logging in, and redirect to a Bitten URL so that slave doesn't need non-build permissions. Closes #459.
osimons
parents: 675
diff changeset
315 resp = self.request('POST', login_url)
648
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
316 match = FORM_TOKEN_RE.search(resp.read())
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
317 if not match:
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
318 log.error("Project %s does not support form "
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
319 "authentication" % url[:-7])
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
320 raise ExitSlave(EX_NOPERM)
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
321 values = {'user': self.username,
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
322 'password':
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
323 self.password_mgr.find_user_password(
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
324 None, url)[1],
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
325 'referer': '',
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
326 '__FORM_TOKEN': match.group(1)}
715
cd4405466a77 Add a `referer=` argument when logging in, and redirect to a Bitten URL so that slave doesn't need non-build permissions. Closes #459.
osimons
parents: 675
diff changeset
327 self.request('POST', login_url,
648
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
328 body=urllib.urlencode(values))
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
329 else:
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
330 log.debug('Performing basic/digest authentication')
715
cd4405466a77 Add a `referer=` argument when logging in, and redirect to a Bitten URL so that slave doesn't need non-build permissions. Closes #459.
osimons
parents: 675
diff changeset
331 self.request('HEAD', login_url)
648
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
332 self.auth_map[url] = True
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
333 elif self.username:
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
334 log.debug('Reusing authentication information.')
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
335 else:
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
336 log.debug('Authentication not provided. Attempting to '
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
337 'execute build anonymously.')
494
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents: 490
diff changeset
338 job_done = self._create_build(url)
480
5181353ccb41 Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181
wbell
parents: 474
diff changeset
339 if job_done:
5181353ccb41 Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181
wbell
parents: 474
diff changeset
340 continue
465
04205ba5dc98 handle HTTPErrors separately to work around them not having a "reason" attribute like URLError
mgood
parents: 461
diff changeset
341 except urllib2.HTTPError, e:
04205ba5dc98 handle HTTPErrors separately to work around them not having a "reason" attribute like URLError
mgood
parents: 461
diff changeset
342 # HTTPError doesn't have the "reason" attribute of URLError
04205ba5dc98 handle HTTPErrors separately to work around them not having a "reason" attribute like URLError
mgood
parents: 461
diff changeset
343 log.error(e)
495
7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents: 494
diff changeset
344 raise ExitSlave(EX_UNAVAILABLE)
402
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents: 401
diff changeset
345 except urllib2.URLError, e:
457
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and
jonas
parents: 456
diff changeset
346 # Is this a temporary network glitch or something a bit
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and
jonas
parents: 456
diff changeset
347 # more severe?
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and
jonas
parents: 456
diff changeset
348 if isinstance(e.reason, socket.error) and \
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and
jonas
parents: 456
diff changeset
349 e.reason.args[0] in temp_net_errors:
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and
jonas
parents: 456
diff changeset
350 log.warning(e)
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and
jonas
parents: 456
diff changeset
351 else:
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and
jonas
parents: 456
diff changeset
352 log.error(e)
495
7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents: 494
diff changeset
353 raise ExitSlave(EX_UNAVAILABLE)
490
5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code
mgood
parents: 480
diff changeset
354 except ExitSlave, e:
5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code
mgood
parents: 480
diff changeset
355 return e.exit_code
525
bb448ad9bf35 Added option for not looping after check. Fixes #371
dfraser
parents: 515
diff changeset
356 if self.no_loop:
bb448ad9bf35 Added option for not looping after check. Fixes #371
dfraser
parents: 515
diff changeset
357 break
449
01503ed994e7 Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents: 442
diff changeset
358 time.sleep(self.poll_interval)
127
a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29.
cmlenz
parents: 119
diff changeset
359
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
360 def quit(self):
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
361 log.info('Shutting down')
495
7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents: 494
diff changeset
362 raise ExitSlave(EX_OK)
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
363
494
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents: 490
diff changeset
364 def _create_build(self, url):
649
eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents: 648
diff changeset
365 xml = xmlio.Element('slave', name=self.name, version=PROTOCOL_VERSION)[
233
8f816147620f * Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents: 221
diff changeset
366 xmlio.Element('platform', processor=self.config['processor'])[
8f816147620f * Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents: 221
diff changeset
367 self.config['machine']
8f816147620f * Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents: 221
diff changeset
368 ],
8f816147620f * Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents: 221
diff changeset
369 xmlio.Element('os', family=self.config['family'],
244
1aa624af9ebb * Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents: 242
diff changeset
370 version=self.config['version'])[
233
8f816147620f * Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents: 221
diff changeset
371 self.config['os']
8f816147620f * Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents: 221
diff changeset
372 ],
29
2adc3480e4aa Some cleanup and additional docstrings.
cmlenz
parents: 19
diff changeset
373 ]
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
374
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
375 log.debug('Configured packages: %s', self.config.packages)
233
8f816147620f * Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents: 221
diff changeset
376 for package, properties in self.config.packages.items():
8f816147620f * Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents: 221
diff changeset
377 xml.append(xmlio.Element('package', name=package, **properties))
8f816147620f * Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents: 221
diff changeset
378
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
379 body = str(xml)
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
380 log.debug('Sending slave configuration: %s', body)
494
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents: 490
diff changeset
381 resp = self.request('POST', url, body, {
559
bf3e0c2f6b80 Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents: 549
diff changeset
382 'Content-Length': str(len(body)),
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
383 'Content-Type': 'application/x-bitten+xml'
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
384 })
277
1141027071b3 Changes to snapshot archive creation/transmission:
cmlenz
parents: 244
diff changeset
385
402
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents: 401
diff changeset
386 if resp.code == 201:
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents: 401
diff changeset
387 self._initiate_build(resp.info().get('location'))
480
5181353ccb41 Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181
wbell
parents: 474
diff changeset
388 return True
402
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents: 401
diff changeset
389 elif resp.code == 204:
420
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
390 log.info('No pending builds')
480
5181353ccb41 Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181
wbell
parents: 474
diff changeset
391 return False
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
392 else:
402
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents: 401
diff changeset
393 log.error('Unexpected response (%d %s)', resp.code, resp.msg)
495
7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents: 494
diff changeset
394 raise ExitSlave(EX_PROTOCOL)
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
395
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
396 def _initiate_build(self, build_url):
418
0cfc877405d1 Fix build slave error introduced in [461].
cmlenz
parents: 417
diff changeset
397 log.info('Build pending at %s', build_url)
420
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
398 try:
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
399 resp = self.request('GET', build_url)
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
400 if resp.code == 200:
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
401 self._execute_build(build_url, resp)
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
402 else:
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
403 log.error('Unexpected response (%d): %s', resp.code, resp.msg)
495
7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents: 494
diff changeset
404 self._cancel_build(build_url, exit_code=EX_PROTOCOL)
420
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
405 except KeyboardInterrupt:
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
406 log.warning('Build interrupted')
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
407 self._cancel_build(build_url)
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
408
414
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
409 def _execute_build(self, build_url, fileobj):
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
410 build_id = build_url and int(build_url.split('/')[-1]) or 0
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
411 xml = xmlio.parse(fileobj)
580
c66190747a9c 0.6dev: In the event that loading Recipe fails, `basedir` would not yet have been defined, and removing it would fail.
osimons
parents: 576
diff changeset
412 basedir = ''
414
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
413 try:
785
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
414 if not self.local:
796
031ce03d2baf Don't initialize the keep-alive thread when running a local build (it will fail trying to get the opener).
osimons
parents: 785
diff changeset
415 keepalive_thread = KeepAliveThread(self.opener, build_url,
031ce03d2baf Don't initialize the keep-alive thread when running a local build (it will fail trying to get the opener).
osimons
parents: 785
diff changeset
416 self.single_build, self.keepalive_interval)
785
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
417 keepalive_thread.start()
466
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents: 465
diff changeset
418 recipe = Recipe(xml, os.path.join(self.work_dir, self.build_dir),
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents: 465
diff changeset
419 self.config)
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents: 465
diff changeset
420 basedir = recipe.ctxt.basedir
490
5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code
mgood
parents: 480
diff changeset
421 log.debug('Running build in directory %s' % basedir)
466
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents: 465
diff changeset
422 if not os.path.exists(basedir):
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents: 465
diff changeset
423 os.mkdir(basedir)
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents: 465
diff changeset
424
414
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
425 for step in recipe:
754
545be0c8f405 Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents: 719
diff changeset
426 try:
545be0c8f405 Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents: 719
diff changeset
427 log.info('Executing build step %r, onerror = %s', step.id, step.onerror)
545be0c8f405 Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents: 719
diff changeset
428 if not self._execute_step(build_url, recipe, step):
545be0c8f405 Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents: 719
diff changeset
429 log.warning('Stopping build due to failure')
545be0c8f405 Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents: 719
diff changeset
430 break
545be0c8f405 Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents: 719
diff changeset
431 except Exception, e:
545be0c8f405 Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents: 719
diff changeset
432 log.error('Exception raised processing step %s. Reraising %s', step.id, e)
545be0c8f405 Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents: 719
diff changeset
433 raise
414
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
434 else:
417
ab2557ff0d97 Change the default log verbosity of the build slave, and remove the `--debug` option.
cmlenz
parents: 414
diff changeset
435 log.info('Build completed')
461
f487963dd886 When the slave is run in `--dry-run` mode, it now also sends a build cancellation message to the master in case of failed builds. Closes #187.
cmlenz
parents: 457
diff changeset
436 if self.dry_run:
f487963dd886 When the slave is run in `--dry-run` mode, it now also sends a build cancellation message to the master in case of failed builds. Closes #187.
cmlenz
parents: 457
diff changeset
437 self._cancel_build(build_url)
414
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
438 finally:
796
031ce03d2baf Don't initialize the keep-alive thread when running a local build (it will fail trying to get the opener).
osimons
parents: 785
diff changeset
439 if not self.local:
031ce03d2baf Don't initialize the keep-alive thread when running a local build (it will fail trying to get the opener).
osimons
parents: 785
diff changeset
440 keepalive_thread.stop()
580
c66190747a9c 0.6dev: In the event that loading Recipe fails, `basedir` would not yet have been defined, and removing it would fail.
osimons
parents: 576
diff changeset
441 if not self.keep_files and os.path.isdir(basedir):
414
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
442 log.debug('Removing build directory %s' % basedir)
474
8c3dfe0efe78 Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents: 466
diff changeset
443 _rmtree(basedir)
414
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
444 if self.single_build:
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
445 log.info('Exiting after single build completed.')
495
7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents: 494
diff changeset
446 raise ExitSlave(EX_OK)
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
447
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
448 def _execute_step(self, build_url, recipe, step):
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
449 failed = False
758
d11ef8024d7c Make all times be generated by the server. This eliminates a number of inconsistencies you'll see when the clocks on the slaves are skewed from the master.
wbell
parents: 754
diff changeset
450 started = int(time.time())
d11ef8024d7c Make all times be generated by the server. This eliminates a number of inconsistencies you'll see when the clocks on the slaves are skewed from the master.
wbell
parents: 754
diff changeset
451 xml = xmlio.Element('result', step=step.id)
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
452 try:
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
453 for type, category, generator, output in \
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
454 step.execute(recipe.ctxt):
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
455 if type == Recipe.ERROR:
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
456 failed = True
442
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
457 if type == Recipe.REPORT and self.dump_reports:
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
458 print output
835
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
459 if type == Recipe.ATTACH:
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
460 # Attachments are added out-of-band due to major
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
461 # performance issues with inlined base64 xml content
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
462 self._attach_file(build_url, recipe, output)
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
463 xml.append(xmlio.Element(type, category=category,
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
464 generator=generator)[
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
465 output
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
466 ])
414
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents: 411
diff changeset
467 except KeyboardInterrupt:
417
ab2557ff0d97 Change the default log verbosity of the build slave, and remove the `--debug` option.
cmlenz
parents: 414
diff changeset
468 log.warning('Build interrupted')
420
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
469 self._cancel_build(build_url)
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
470 except BuildError, e:
628
05686657e989 0.6dev: Implement a `subprocess` implementation of `execute()` that is now the default for all slaves where this is available - essentially all slaves with Python 2.4 and higher, or where module is installed separately. This fixes:
osimons
parents: 610
diff changeset
471 log.error('Build step %r failed', step.id)
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
472 failed = True
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
473 except Exception, e:
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
474 log.error('Internal error in build step %r', step.id, exc_info=True)
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
475 failed = True
758
d11ef8024d7c Make all times be generated by the server. This eliminates a number of inconsistencies you'll see when the clocks on the slaves are skewed from the master.
wbell
parents: 754
diff changeset
476 xml.attr['duration'] = (time.time() - started)
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
477 if failed:
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
478 xml.attr['status'] = 'failure'
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
479 else:
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
480 xml.attr['status'] = 'success'
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
481 log.info('Build step %s completed successfully', step.id)
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
482
424
bc81d7b069ff Rudimentary implementation of ``dry-run`` option in build slave. Builds are still put in in progress mode, but no results are transmitted back, and the build is automatically invalidated after the slave is done.
cmlenz
parents: 420
diff changeset
483 if not self.local and not self.dry_run:
420
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
484 try:
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
485 resp = self.request('POST', build_url + '/steps/', str(xml), {
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
486 'Content-Type': 'application/x-bitten+xml'
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
487 })
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
488 if resp.code != 201:
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
489 log.error('Unexpected response (%d): %s', resp.code,
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
490 resp.msg)
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
491 except KeyboardInterrupt:
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
492 log.warning('Build interrupted')
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
493 self._cancel_build(build_url)
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
494 return not failed or step.onerror != 'fail'
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
495
495
7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents: 494
diff changeset
496 def _cancel_build(self, build_url, exit_code=EX_OK):
420
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
497 log.info('Cancelling build at %s', build_url)
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
498 if not self.local:
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
499 resp = self.request('DELETE', build_url)
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
500 if resp.code not in (200, 204):
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
501 log.error('Unexpected response (%d): %s', resp.code, resp.msg)
490
5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code
mgood
parents: 480
diff changeset
502 raise ExitSlave(exit_code)
420
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents: 418
diff changeset
503
835
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
504 def _attach_file(self, build_url, recipe, attachment):
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
505 form_token = recipe._root.attr.get('form_token', '')
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
506 if self.local or self.dry_run or not form_token:
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
507 log.info('Attachment %s not sent due to current slave options',
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
508 attachment.attr['file'])
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
509 return
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
510 resource_type = attachment.attr['resource']
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
511 url = str(build_url + '/attach/' + resource_type)
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
512 path = recipe.ctxt.resolve(attachment.attr['filename'])
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
513 filename = os.path.basename(path)
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
514 log.debug('Attaching file %s to %s...', attachment.attr['filename'],
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
515 resource_type)
889
9ac91d38198c Make sure slave reads attachments as binary files. Fixes #657.
osimons
parents: 870
diff changeset
516 f = open(path, 'rb')
835
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
517 try:
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
518 data, content_type = encode_multipart_formdata({
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
519 'file': (filename, f.read()),
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
520 'description': attachment.attr['description'],
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
521 '__FORM_TOKEN': form_token})
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
522 finally:
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
523 f.close()
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
524 resp = self.request('POST', url , data, {
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
525 'Content-Type': content_type})
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
526 if not resp.code == 201:
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
527 msg = 'Error attaching %s to %s'
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
528 log.error(msg, filename, resource_type)
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents: 832
diff changeset
529 raise BuildError(msg, filename, resource_type)
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
530
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
531 class ExitSlave(Exception):
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
532 """Exception used internally by the slave to signal that the slave process
411
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
533 should be stopped.
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
534 """
490
5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code
mgood
parents: 480
diff changeset
535 def __init__(self, exit_code):
5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code
mgood
parents: 480
diff changeset
536 self.exit_code = exit_code
5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code
mgood
parents: 480
diff changeset
537 Exception.__init__(self)
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
538
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
539
31
0b82d012e42c Add thin script shells around master and slave.
cmlenz
parents: 29
diff changeset
540 def main():
313
90422699a594 More and improved docstrings (using epydoc format).
cmlenz
parents: 310
diff changeset
541 """Main entry point for running the build slave."""
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
542 from bitten import __version__ as VERSION
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
543 from optparse import OptionParser
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
544
494
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents: 490
diff changeset
545 parser = OptionParser(usage='usage: %prog [options] url1 [url2] ...',
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
546 version='%%prog %s' % VERSION)
185
2c24d9a950ed Add a `--dry-run` option to the build slave. This will result in the slave being registered and executing builds, but without submitting the progress and results of the build back to the server. Useful for getting the configuration of new slaves right without polluting the database with invalid builds.
cmlenz
parents: 163
diff changeset
547 parser.add_option('--name', action='store', dest='name',
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
548 help='name of this slave (defaults to host name)')
208
f39136272683 * Slave now removes build working directories when done.
cmlenz
parents: 203
diff changeset
549 parser.add_option('-f', '--config', action='store', dest='config',
127
a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29.
cmlenz
parents: 119
diff changeset
550 metavar='FILE', help='path to configuration file')
402
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents: 401
diff changeset
551 parser.add_option('-u', '--user', dest='username',
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents: 401
diff changeset
552 help='the username to use for authentication')
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents: 401
diff changeset
553 parser.add_option('-p', '--password', dest='password',
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents: 401
diff changeset
554 help='the password to use when authenticating')
587
0ac21e3343a4 0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents: 580
diff changeset
555 def _ask_password(option, opt_str, value, parser):
0ac21e3343a4 0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents: 580
diff changeset
556 from getpass import getpass
0ac21e3343a4 0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents: 580
diff changeset
557 parser.values.password = getpass('Passsword: ')
0ac21e3343a4 0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents: 580
diff changeset
558 parser.add_option('-P', '--ask-password', action='callback',
0ac21e3343a4 0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents: 580
diff changeset
559 callback=_ask_password, help='Prompt for password')
648
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
560 parser.add_option('--form-auth', action='store_true',
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
561 dest='form_auth',
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
562 help='login using AccountManager HTML form instead of '
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
563 'HTTP authentication for all urls')
442
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
564
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
565 group = parser.add_option_group('building')
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
566 group.add_option('-d', '--work-dir', action='store', dest='work_dir',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
567 metavar='DIR', help='working directory for builds')
466
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents: 465
diff changeset
568 group.add_option('--build-dir', action='store', dest='build_dir',
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents: 465
diff changeset
569 default = 'build_${config}_${build}',
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents: 465
diff changeset
570 help='name pattern for the build dir to use inside the '
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents: 465
diff changeset
571 'working dir ["%default"]')
442
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
572 group.add_option('-k', '--keep-files', action='store_true',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
573 dest='keep_files',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
574 help='don\'t delete files after builds')
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
575 group.add_option('-s', '--single', action='store_true',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
576 dest='single_build',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
577 help='exit after completing a single build')
525
bb448ad9bf35 Added option for not looping after check. Fixes #371
dfraser
parents: 515
diff changeset
578 group.add_option('', '--no-loop', action='store_true',
bb448ad9bf35 Added option for not looping after check. Fixes #371
dfraser
parents: 515
diff changeset
579 dest='no_loop',
bb448ad9bf35 Added option for not looping after check. Fixes #371
dfraser
parents: 515
diff changeset
580 help='exit after completing a single check and running '
bb448ad9bf35 Added option for not looping after check. Fixes #371
dfraser
parents: 515
diff changeset
581 'the required builds')
442
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
582 group.add_option('-n', '--dry-run', action='store_true', dest='dry_run',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
583 help='don\'t report results back to master')
449
01503ed994e7 Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents: 442
diff changeset
584 group.add_option('-i', '--interval', dest='interval', metavar='SECONDS',
01503ed994e7 Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents: 442
diff changeset
585 type='int', help='time to wait between requesting builds')
785
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
586 group.add_option('-b', '--keepalive_interval', dest='keepalive_interval', metavar='SECONDS', type='int', help='time to wait between keepalive heartbeats')
442
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
587 group = parser.add_option_group('logging')
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
588 group.add_option('-l', '--log', dest='logfile', metavar='FILENAME',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
589 help='write log messages to FILENAME')
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
590 group.add_option('-v', '--verbose', action='store_const', dest='loglevel',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
591 const=logging.DEBUG, help='print as much as possible')
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
592 group.add_option('-q', '--quiet', action='store_const', dest='loglevel',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
593 const=logging.WARN, help='print as little as possible')
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
594 group.add_option('--dump-reports', action='store_true', dest='dump_reports',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
595 help='whether report data should be printed')
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
596
242
372d1de2e3ec * Fixes to the `<c:configure>` command added in [247]: Set current directory when invoking the script, and correctly pass `CFLAGS` and `CXXFLAGS`.
cmlenz
parents: 241
diff changeset
597 parser.set_defaults(dry_run=False, keep_files=False,
525
bb448ad9bf35 Added option for not looping after check. Fixes #371
dfraser
parents: 515
diff changeset
598 loglevel=logging.INFO, single_build=False, no_loop=False,
785
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
599 dump_reports=False, interval=300, keepalive_interval=60,
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
600 form_auth=False)
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
601 options, args = parser.parse_args()
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
602
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
603 if len(args) < 1:
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
604 parser.error('incorrect number of arguments')
494
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents: 490
diff changeset
605 urls = args
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
606
157
2efdc69e63c3 Some style/documentation improvements to make Pylint happier.
cmlenz
parents: 149
diff changeset
607 logger = logging.getLogger('bitten')
2efdc69e63c3 Some style/documentation improvements to make Pylint happier.
cmlenz
parents: 149
diff changeset
608 logger.setLevel(options.loglevel)
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 90
diff changeset
609 handler = logging.StreamHandler()
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 90
diff changeset
610 handler.setLevel(options.loglevel)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 90
diff changeset
611 formatter = logging.Formatter('[%(levelname)-8s] %(message)s')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 90
diff changeset
612 handler.setFormatter(formatter)
157
2efdc69e63c3 Some style/documentation improvements to make Pylint happier.
cmlenz
parents: 149
diff changeset
613 logger.addHandler(handler)
339
6469c692c603 Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents: 318
diff changeset
614 if options.logfile:
6469c692c603 Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents: 318
diff changeset
615 handler = logging.FileHandler(options.logfile)
6469c692c603 Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents: 318
diff changeset
616 handler.setLevel(options.loglevel)
6469c692c603 Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents: 318
diff changeset
617 formatter = logging.Formatter('%(asctime)s [%(name)s] %(levelname)s: '
6469c692c603 Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents: 318
diff changeset
618 '%(message)s')
6469c692c603 Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents: 318
diff changeset
619 handler.setFormatter(formatter)
6469c692c603 Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents: 318
diff changeset
620 logger.addHandler(handler)
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
621
610
610a97f7e1ba 0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents: 587
diff changeset
622 log.info("Slave launched at %s" % \
610a97f7e1ba 0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents: 587
diff changeset
623 datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
610a97f7e1ba 0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents: 587
diff changeset
624
675
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents: 654
diff changeset
625 slave = None
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents: 654
diff changeset
626 try:
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents: 654
diff changeset
627 slave = BuildSlave(urls, name=options.name, config=options.config,
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
628 dry_run=options.dry_run, work_dir=options.work_dir,
466
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents: 465
diff changeset
629 build_dir=options.build_dir,
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
630 keep_files=options.keep_files,
402
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents: 401
diff changeset
631 single_build=options.single_build,
543
ce16a5dcf8b3 Pass `--no-loop` option to the `BuildSlave` (fixes #375, from the patch there - thanks!)
dfraser
parents: 542
diff changeset
632 no_loop=options.no_loop,
449
01503ed994e7 Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents: 442
diff changeset
633 poll_interval=options.interval,
785
8e76b8f6310a Adding keepalives to the bitten client/server protocol.
wbell
parents: 758
diff changeset
634 keepalive_interval=options.keepalive_interval,
442
a8787de4fbc3 Improve the still experimental support for using `coverage.py`.
cmlenz
parents: 436
diff changeset
635 username=options.username, password=options.password,
648
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
636 dump_reports=options.dump_reports,
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents: 647
diff changeset
637 form_auth=options.form_auth)
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
638 try:
490
5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code
mgood
parents: 480
diff changeset
639 exit_code = slave.run()
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
640 except KeyboardInterrupt:
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 387
diff changeset
641 slave.quit()
675
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents: 654
diff changeset
642 except ConfigFileNotFound, e:
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents: 654
diff changeset
643 log.error(e)
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents: 654
diff changeset
644 exit_code = EX_IOERR
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents: 654
diff changeset
645 except MissingSectionHeaderError:
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents: 654
diff changeset
646 log.error("Error parsing configuration file %r. Wrong format?" \
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents: 654
diff changeset
647 % options.config)
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents: 654
diff changeset
648 exit_code = EX_IOERR
490
5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code
mgood
parents: 480
diff changeset
649 except ExitSlave, e:
5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code
mgood
parents: 480
diff changeset
650 exit_code = e.exit_code
31
0b82d012e42c Add thin script shells around master and slave.
cmlenz
parents: 29
diff changeset
651
675
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents: 654
diff changeset
652 if slave and not (options.work_dir or options.keep_files):
576
4b75e3f6f27c 0.6dev: Fixing #229 again - keeping `slave.work_dir` if either `--keep_files` or `--work_dir=` are part of options.
osimons
parents: 570
diff changeset
653 log.debug('Removing working directory %s' % slave.work_dir)
474
8c3dfe0efe78 Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents: 466
diff changeset
654 _rmtree(slave.work_dir)
610
610a97f7e1ba 0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents: 587
diff changeset
655
610a97f7e1ba 0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents: 587
diff changeset
656 log.info("Slave exited at %s" % \
610a97f7e1ba 0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents: 587
diff changeset
657 datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
610a97f7e1ba 0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents: 587
diff changeset
658
490
5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code
mgood
parents: 480
diff changeset
659 return exit_code
208
f39136272683 * Slave now removes build working directories when done.
cmlenz
parents: 203
diff changeset
660
31
0b82d012e42c Add thin script shells around master and slave.
cmlenz
parents: 29
diff changeset
661 if __name__ == '__main__':
490
5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code
mgood
parents: 480
diff changeset
662 sys.exit(main())
Copyright (C) 2012-2017 Edgewall Software