Mercurial > bitten > bitten-test
annotate bitten/master.py @ 755:ab91df14f670 0.6.x
Merge of [832] from trunk.
author | wbell |
---|---|
date | Sat, 24 Apr 2010 13:38:25 +0000 |
parents | db5f7388535d |
children | bfd5fd75d1d6 |
rev | line source |
---|---|
379 | 1 # -*- coding: utf-8 -*- |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
2 # |
408
933105ab516b
Update file headers and other stuff pointing to the old home.
cmlenz
parents:
402
diff
changeset
|
3 # Copyright (C) 2007 Edgewall Software |
933105ab516b
Update file headers and other stuff pointing to the old home.
cmlenz
parents:
402
diff
changeset
|
4 # Copyright (C) 2005-2007 Christopher Lenz <cmlenz@gmx.de> |
163 | 5 # All rights reserved. |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
6 # |
163 | 7 # This software is licensed as described in the file COPYING, which |
8 # you should have received as part of this distribution. The terms | |
408
933105ab516b
Update file headers and other stuff pointing to the old home.
cmlenz
parents:
402
diff
changeset
|
9 # are also available at http://bitten.edgewall.org/wiki/License. |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
10 |
392 | 11 """Build master implementation.""" |
313 | 12 |
312 | 13 import calendar |
392 | 14 import re |
47
083e848088ee
* Improvements to the model classes, and a couple of unit tests.
cmlenz
parents:
45
diff
changeset
|
15 import time |
629
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
16 from StringIO import StringIO |
18
591a5a836ecc
* {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents:
15
diff
changeset
|
17 |
629
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
18 from trac.attachment import Attachment |
577
dcee8ff20e82
0.6dev: The `logs_dir` option is not a real `PathOption` as for that it would need to support config-relative path with possibly chained configs following trac:ticket:8358 (for 0.11.5). The fix just makes it a regular `Option` to keep current behaviour for all Trac 0.11+ versions.
osimons
parents:
568
diff
changeset
|
19 from trac.config import BoolOption, IntOption, Option |
392 | 20 from trac.core import * |
629
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
21 from trac.resource import ResourceNotFound |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
22 from trac.web import IRequestHandler, RequestDone |
392 | 23 |
654
cc1fefd5d814
0.6dev: Moved a constant declaration so that master should still work with Python 2.4. Updated docs with regards to Python requirements + added a changelog message from cmlenz.
osimons
parents:
649
diff
changeset
|
24 from bitten import PROTOCOL_VERSION |
554
2c27f3581100
Supply the target platform as a variable `platform` for build recipes, and adjust test accordingly - fixes #376
dfraser
parents:
542
diff
changeset
|
25 from bitten.model import BuildConfig, Build, BuildStep, BuildLog, Report, \ |
2c27f3581100
Supply the target platform as a variable `platform` for build recipes, and adjust test accordingly - fixes #376
dfraser
parents:
542
diff
changeset
|
26 TargetPlatform |
2c27f3581100
Supply the target platform as a variable `platform` for build recipes, and adjust test accordingly - fixes #376
dfraser
parents:
542
diff
changeset
|
27 |
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
28 from bitten.main import BuildSystem |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
29 from bitten.queue import BuildQueue |
392 | 30 from bitten.recipe import Recipe |
31 from bitten.util import xmlio | |
83
42970c14524a
Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents:
82
diff
changeset
|
32 |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
410
diff
changeset
|
33 __all__ = ['BuildMaster'] |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
410
diff
changeset
|
34 __docformat__ = 'restructuredtext en' |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
410
diff
changeset
|
35 |
56 | 36 |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
37 HTTP_BAD_REQUEST = 400 |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
38 HTTP_FORBIDDEN = 403 |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
39 HTTP_NOT_FOUND = 404 |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
40 HTTP_METHOD_NOT_ALLOWED = 405 |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
41 HTTP_CONFLICT = 409 |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
42 |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
43 |
392 | 44 class BuildMaster(Component): |
542 | 45 """Trac request handler implementation for the build master.""" |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
46 |
392 | 47 implements(IRequestHandler) |
51
5caccd7b247e
Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents:
49
diff
changeset
|
48 |
392 | 49 # Configuration options |
51
5caccd7b247e
Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents:
49
diff
changeset
|
50 |
392 | 51 adjust_timestamps = BoolOption('bitten', 'adjust_timestamps', False, doc= |
432 | 52 """Whether the timestamps of builds should be adjusted to be close |
392 | 53 to the timestamps of the corresponding changesets.""") |
54 | |
55 build_all = BoolOption('bitten', 'build_all', False, doc= | |
56 """Whether to request builds of older revisions even if a younger | |
57 revision has already been built.""") | |
468
44c2b4ac6157
Add stabilization time parameter to build master. Closes #189. Many thanks to Allen Bierbaum for the patch.
cmlenz
parents:
466
diff
changeset
|
58 |
44c2b4ac6157
Add stabilization time parameter to build master. Closes #189. Many thanks to Allen Bierbaum for the patch.
cmlenz
parents:
466
diff
changeset
|
59 stabilize_wait = IntOption('bitten', 'stabilize_wait', 0, doc= |
44c2b4ac6157
Add stabilization time parameter to build master. Closes #189. Many thanks to Allen Bierbaum for the patch.
cmlenz
parents:
466
diff
changeset
|
60 """The time in seconds to wait for the repository to stabilize before |
44c2b4ac6157
Add stabilization time parameter to build master. Closes #189. Many thanks to Allen Bierbaum for the patch.
cmlenz
parents:
466
diff
changeset
|
61 queuing up a new build. This allows time for developers to check in |
44c2b4ac6157
Add stabilization time parameter to build master. Closes #189. Many thanks to Allen Bierbaum for the patch.
cmlenz
parents:
466
diff
changeset
|
62 a group of related changes back to back without spawning multiple |
44c2b4ac6157
Add stabilization time parameter to build master. Closes #189. Many thanks to Allen Bierbaum for the patch.
cmlenz
parents:
466
diff
changeset
|
63 builds.""") |
392 | 64 |
65 slave_timeout = IntOption('bitten', 'slave_timeout', 3600, doc= | |
66 """The time in seconds after which a build is cancelled if the slave | |
67 does not report progress.""") | |
68 | |
577
dcee8ff20e82
0.6dev: The `logs_dir` option is not a real `PathOption` as for that it would need to support config-relative path with possibly chained configs following trac:ticket:8358 (for 0.11.5). The fix just makes it a regular `Option` to keep current behaviour for all Trac 0.11+ versions.
osimons
parents:
568
diff
changeset
|
69 logs_dir = Option('bitten', 'logs_dir', "log/bitten", doc= |
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
494
diff
changeset
|
70 """The directory on the server in which client log files will be stored.""") |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
494
diff
changeset
|
71 |
557
b4d3d9cbf200
Alter the appearance of the ''Build Status'' button, to show the current build status [eblot] - fixes #373
dfraser
parents:
554
diff
changeset
|
72 quick_status = BoolOption('bitten', 'quick_status', False, doc= |
b4d3d9cbf200
Alter the appearance of the ''Build Status'' button, to show the current build status [eblot] - fixes #373
dfraser
parents:
554
diff
changeset
|
73 """Whether to show the current build status withing the Trac main |
b4d3d9cbf200
Alter the appearance of the ''Build Status'' button, to show the current build status [eblot] - fixes #373
dfraser
parents:
554
diff
changeset
|
74 navigation bar""") |
b4d3d9cbf200
Alter the appearance of the ''Build Status'' button, to show the current build status [eblot] - fixes #373
dfraser
parents:
554
diff
changeset
|
75 |
568 | 76 def __init__(self): |
77 self.env.systeminfo.append(('Bitten', | |
78 __import__('bitten', ['__version__']).__version__)) | |
79 | |
392 | 80 # IRequestHandler methods |
81 | |
82 def match_request(self, req): | |
401
a10942252ebc
Use POST for submitting step results, instead of PUT.
cmlenz
parents:
392
diff
changeset
|
83 match = re.match(r'/builds(?:/(\d+)(?:/(\w+)/([^/]+)?)?)?$', |
392 | 84 req.path_info) |
85 if match: | |
86 if match.group(1): | |
87 req.args['id'] = match.group(1) | |
88 req.args['collection'] = match.group(2) | |
89 req.args['member'] = match.group(3) | |
90 return True | |
91 | |
92 def process_request(self, req): | |
93 req.perm.assert_permission('BUILD_EXEC') | |
94 | |
649
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
95 if 'trac_auth' in req.incookie: |
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
96 slave_token = req.incookie['trac_auth'].value |
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
97 else: |
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
98 slave_token = req.session.sid |
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
99 |
392 | 100 if 'id' not in req.args: |
101 if req.method != 'POST': | |
716 | 102 self._send_response(req, |
103 body='Only POST allowed for build creation.') | |
649
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
104 return self._process_build_creation(req, slave_token) |
392 | 105 |
106 build = Build.fetch(self.env, req.args['id']) | |
107 if not build: | |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
108 self._send_error(req, HTTP_NOT_FOUND, |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
109 'No such build (%s)' % req.args['id']) |
649
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
110 |
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
111 build_token = build.slave_info.get('token', '') |
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
112 if build_token != slave_token: |
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
113 self._send_error(req, HTTP_CONFLICT, |
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
114 'Token mismatch (wrong slave): slave=%s, build=%s' \ |
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
115 % (slave_token, build_token)) |
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
116 |
392 | 117 config = BuildConfig.fetch(self.env, build.config) |
118 | |
119 if not req.args['collection']: | |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
120 if req.method == 'DELETE': |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
121 return self._process_build_cancellation(req, config, build) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
122 else: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
123 return self._process_build_initiation(req, config, build) |
392 | 124 |
401
a10942252ebc
Use POST for submitting step results, instead of PUT.
cmlenz
parents:
392
diff
changeset
|
125 if req.method != 'POST': |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
126 self._send_error(req, HTTP_METHOD_NOT_ALLOWED, |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
127 'Method %s not allowed' % req.method) |
392 | 128 |
129 if req.args['collection'] == 'steps': | |
401
a10942252ebc
Use POST for submitting step results, instead of PUT.
cmlenz
parents:
392
diff
changeset
|
130 return self._process_build_step(req, config, build) |
392 | 131 else: |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
132 self._send_error(req, HTTP_NOT_FOUND, |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
133 "No such collection '%s'" % req.args['collection']) |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
134 |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
135 # Internal methods |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
136 |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
137 def _send_response(self, req, code=200, body='', headers=None): |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
138 """ Formats and sends the response, raising ``RequestDone``. """ |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
139 req.send_response(code) |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
140 headers = headers or {} |
703 | 141 headers.setdefault('Content-Length', len(body)) |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
142 for header in headers: |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
143 req.send_header(header, headers[header]) |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
144 req.write(body) |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
145 raise RequestDone |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
146 |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
147 def _send_error(self, req, code=500, message=''): |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
148 """ Formats and sends the error, raising ``RequestDone``. """ |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
149 headers = {'Content-Type': 'text/plain', |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
150 'Content-Length': str(len(message))} |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
151 self._send_response(req, code, body=message, headers=headers) |
392 | 152 |
649
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
153 def _process_build_creation(self, req, slave_token): |
468
44c2b4ac6157
Add stabilization time parameter to build master. Closes #189. Many thanks to Allen Bierbaum for the patch.
cmlenz
parents:
466
diff
changeset
|
154 queue = BuildQueue(self.env, build_all=self.build_all, |
44c2b4ac6157
Add stabilization time parameter to build master. Closes #189. Many thanks to Allen Bierbaum for the patch.
cmlenz
parents:
466
diff
changeset
|
155 stabilize_wait=self.stabilize_wait, |
419
b72802dc0632
Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents:
411
diff
changeset
|
156 timeout=self.slave_timeout) |
392 | 157 queue.populate() |
158 | |
159 try: | |
160 elem = xmlio.parse(req.read()) | |
161 except xmlio.ParseError, e: | |
492
56f0ad35c60a
Log errors when build master encounters an XML parse error.
cmlenz
parents:
479
diff
changeset
|
162 self.log.error('Error parsing build initialization request: %s', e, |
56f0ad35c60a
Log errors when build master encounters an XML parse error.
cmlenz
parents:
479
diff
changeset
|
163 exc_info=True) |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
164 self._send_error(req, HTTP_BAD_REQUEST, 'XML parser error') |
392 | 165 |
649
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
166 slave_version = int(elem.attr.get('version', 1)) |
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
167 if slave_version != PROTOCOL_VERSION: |
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
168 self._send_error(req, HTTP_BAD_REQUEST, |
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
169 "Master-Slave version mismatch: master=%d, slave=%d" % \ |
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
170 (PROTOCOL_VERSION, slave_version)) |
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
171 |
426 | 172 slavename = elem.attr['name'] |
649
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
173 properties = {'name': slavename, Build.IP_ADDRESS: req.remote_addr, |
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
174 Build.TOKEN: slave_token} |
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
175 self.log.info('Build slave %r connected from %s with token %s', |
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
645
diff
changeset
|
176 slavename, req.remote_addr, slave_token) |
392 | 177 |
178 for child in elem.children(): | |
179 if child.name == 'platform': | |
180 properties[Build.MACHINE] = child.gettext() | |
181 properties[Build.PROCESSOR] = child.attr.get('processor') | |
182 elif child.name == 'os': | |
183 properties[Build.OS_NAME] = child.gettext() | |
184 properties[Build.OS_FAMILY] = child.attr.get('family') | |
185 properties[Build.OS_VERSION] = child.attr.get('version') | |
186 elif child.name == 'package': | |
187 for name, value in child.attr.items(): | |
188 if name == 'name': | |
189 continue | |
190 properties[child.attr['name'] + '.' + name] = value | |
191 | |
444
22d6a7da8777
Another minor improvement to logging in the build master.
cmlenz
parents:
436
diff
changeset
|
192 self.log.debug('Build slave configuration: %r', properties) |
22d6a7da8777
Another minor improvement to logging in the build master.
cmlenz
parents:
436
diff
changeset
|
193 |
426 | 194 build = queue.get_build_for_slave(slavename, properties) |
392 | 195 if not build: |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
196 self._send_response(req, 204, '', {}) |
392 | 197 |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
198 self._send_response(req, 201, 'Build pending', headers={ |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
199 'Content-Type': 'text/plain', |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
200 'Location': req.abs_href.builds(build.id)}) |
392 | 201 |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
202 def _process_build_cancellation(self, req, config, build): |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
203 self.log.info('Build slave %r cancelled build %d', build.slave, |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
204 build.id) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
205 build.status = Build.PENDING |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
206 build.slave = None |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
207 build.slave_info = {} |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
208 build.started = 0 |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
209 db = self.env.get_db_cnx() |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
210 for step in list(BuildStep.select(self.env, build=build.id, db=db)): |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
211 step.delete(db=db) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
212 build.update(db=db) |
751
db5f7388535d
Merge [821,822,823,824,825,826] from trunk. Thanks Walter\!
hodgestar
parents:
716
diff
changeset
|
213 |
db5f7388535d
Merge [821,822,823,824,825,826] from trunk. Thanks Walter\!
hodgestar
parents:
716
diff
changeset
|
214 Attachment.delete_all(self.env, 'build', build.resource.id, db) |
db5f7388535d
Merge [821,822,823,824,825,826] from trunk. Thanks Walter\!
hodgestar
parents:
716
diff
changeset
|
215 |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
216 db.commit() |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
217 |
458 | 218 for listener in BuildSystem(self.env).listeners: |
219 listener.build_aborted(build) | |
220 | |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
221 self._send_response(req, 204, '', {}) |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
222 |
392 | 223 def _process_build_initiation(self, req, config, build): |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
224 self.log.info('Build slave %r initiated build %d', build.slave, |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
419
diff
changeset
|
225 build.id) |
392 | 226 build.started = int(time.time()) |
227 build.update() | |
277 | 228 |
458 | 229 for listener in BuildSystem(self.env).listeners: |
230 listener.build_started(build) | |
231 | |
287
6abd43d0cd8a
The build slave now stores snapshot archives and the corresponding work directories in project folders of the main work folder, to keep build configurations from different projects that share the same name separate. This also requires transmitting the project name (simply the name of the environment directory) with the build initiation.
cmlenz
parents:
284
diff
changeset
|
232 xml = xmlio.parse(config.recipe) |
392 | 233 xml.attr['path'] = config.path |
234 xml.attr['revision'] = build.rev | |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
463
diff
changeset
|
235 xml.attr['config'] = config.name |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
463
diff
changeset
|
236 xml.attr['build'] = str(build.id) |
554
2c27f3581100
Supply the target platform as a variable `platform` for build recipes, and adjust test accordingly - fixes #376
dfraser
parents:
542
diff
changeset
|
237 target_platform = TargetPlatform.fetch(self.env, build.platform) |
2c27f3581100
Supply the target platform as a variable `platform` for build recipes, and adjust test accordingly - fixes #376
dfraser
parents:
542
diff
changeset
|
238 xml.attr['platform'] = target_platform.name |
611
294641e84e89
0.6dev: Adding `${name}` and `${basedir}` (#325) for recipe substitution. Updated docs + new test.
osimons
parents:
577
diff
changeset
|
239 xml.attr['name'] = build.slave |
392 | 240 body = str(xml) |
374
446092a2d2fe
Don't accept build results from a slave if the build has been invalidated or it's being built by another slave. Closes #100.
wbell
parents:
373
diff
changeset
|
241 |
473
b1d346c4e539
Only accept build results from the slave that's supposed to be processing this build. Additional logging.
wbell
parents:
468
diff
changeset
|
242 self.log.info('Build slave %r initiated build %d', build.slave, |
b1d346c4e539
Only accept build results from the slave that's supposed to be processing this build. Additional logging.
wbell
parents:
468
diff
changeset
|
243 build.id) |
b1d346c4e539
Only accept build results from the slave that's supposed to be processing this build. Additional logging.
wbell
parents:
468
diff
changeset
|
244 |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
245 self._send_response(req, 200, body, headers={ |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
246 'Content-Type': 'application/x-bitten+xml', |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
247 'Content-Length': str(len(body)), |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
248 'Content-Disposition': |
392 | 249 'attachment; filename=recipe_%s_r%s.xml' % |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
250 (config.name, build.rev)}) |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
251 |
401
a10942252ebc
Use POST for submitting step results, instead of PUT.
cmlenz
parents:
392
diff
changeset
|
252 def _process_build_step(self, req, config, build): |
a10942252ebc
Use POST for submitting step results, instead of PUT.
cmlenz
parents:
392
diff
changeset
|
253 try: |
a10942252ebc
Use POST for submitting step results, instead of PUT.
cmlenz
parents:
392
diff
changeset
|
254 elem = xmlio.parse(req.read()) |
a10942252ebc
Use POST for submitting step results, instead of PUT.
cmlenz
parents:
392
diff
changeset
|
255 except xmlio.ParseError, e: |
492
56f0ad35c60a
Log errors when build master encounters an XML parse error.
cmlenz
parents:
479
diff
changeset
|
256 self.log.error('Error parsing build step result: %s', e, |
56f0ad35c60a
Log errors when build master encounters an XML parse error.
cmlenz
parents:
479
diff
changeset
|
257 exc_info=True) |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
258 self._send_error(req, HTTP_BAD_REQUEST, 'XML parser error') |
401
a10942252ebc
Use POST for submitting step results, instead of PUT.
cmlenz
parents:
392
diff
changeset
|
259 stepname = elem.attr['step'] |
632
01c9848950d5
0.6dev: Cleaning all usage of tabs (replaced with spaces).
osimons
parents:
629
diff
changeset
|
260 |
392 | 261 step = BuildStep.fetch(self.env, build=build.id, name=stepname) |
262 if step: | |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
263 self._send_error(req, HTTP_CONFLICT, 'Build step already exists') |
253 | 264 |
392 | 265 recipe = Recipe(xmlio.parse(config.recipe)) |
266 index = None | |
267 current_step = None | |
268 for num, recipe_step in enumerate(recipe): | |
269 if recipe_step.id == stepname: | |
270 index = num | |
271 current_step = recipe_step | |
272 if index is None: | |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
273 self._send_error(req, HTTP_FORBIDDEN, |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
274 'No such build step' % stepname) |
392 | 275 last_step = index == num |
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
197
diff
changeset
|
276 |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
492
diff
changeset
|
277 self.log.debug('Slave %s (build %d) completed step %d (%s) with ' |
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
492
diff
changeset
|
278 'status %s', build.slave, build.id, index, stepname, |
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
492
diff
changeset
|
279 elem.attr['status']) |
392 | 280 |
281 db = self.env.get_db_cnx() | |
282 | |
283 step = BuildStep(self.env, build=build.id, name=stepname) | |
284 try: | |
285 step.started = int(_parse_iso_datetime(elem.attr['time'])) | |
286 step.stopped = step.started + float(elem.attr['duration']) | |
287 except ValueError, e: | |
492
56f0ad35c60a
Log errors when build master encounters an XML parse error.
cmlenz
parents:
479
diff
changeset
|
288 self.log.error('Error parsing build step timestamp: %s', e, |
56f0ad35c60a
Log errors when build master encounters an XML parse error.
cmlenz
parents:
479
diff
changeset
|
289 exc_info=True) |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
290 self._send_error(req, HTTP_BAD_REQUEST, e.args[0]) |
392 | 291 if elem.attr['status'] == 'failure': |
292 self.log.warning('Build %s step %s failed', build.id, stepname) | |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
293 step.status = BuildStep.FAILURE |
459 | 294 if current_step.onerror == 'fail': |
295 last_step = True | |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
296 else: |
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
297 step.status = BuildStep.SUCCESS |
277 | 298 step.errors += [error.gettext() for error in elem.children('error')] |
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
110
diff
changeset
|
299 step.insert(db=db) |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
300 |
392 | 301 # Collect log messages from the request body |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
302 for idx, log_elem in enumerate(elem.children('log')): |
401
a10942252ebc
Use POST for submitting step results, instead of PUT.
cmlenz
parents:
392
diff
changeset
|
303 build_log = BuildLog(self.env, build=build.id, step=stepname, |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
304 generator=log_elem.attr.get('generator'), |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
305 orderno=idx) |
115
16d69eb6e047
Add support for XML fragments to the {{{xmlio}}} module, so that build output and reports don't need to be nested in a meaningless element (such as {{{<log type="distutils"><messages><message ...>}}}).
cmlenz
parents:
112
diff
changeset
|
306 for message_elem in log_elem.children('message'): |
16d69eb6e047
Add support for XML fragments to the {{{xmlio}}} module, so that build output and reports don't need to be nested in a meaningless element (such as {{{<log type="distutils"><messages><message ...>}}}).
cmlenz
parents:
112
diff
changeset
|
307 build_log.messages.append((message_elem.attr['level'], |
16d69eb6e047
Add support for XML fragments to the {{{xmlio}}} module, so that build output and reports don't need to be nested in a meaningless element (such as {{{<log type="distutils"><messages><message ...>}}}).
cmlenz
parents:
112
diff
changeset
|
308 message_elem.gettext())) |
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
110
diff
changeset
|
309 build_log.insert(db=db) |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
310 |
392 | 311 # Collect report data from the request body |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
312 for report_elem in elem.children('report'): |
401
a10942252ebc
Use POST for submitting step results, instead of PUT.
cmlenz
parents:
392
diff
changeset
|
313 report = Report(self.env, build=build.id, step=stepname, |
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
204
diff
changeset
|
314 category=report_elem.attr.get('category'), |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
204
diff
changeset
|
315 generator=report_elem.attr.get('generator')) |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
316 for item_elem in report_elem.children(): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
317 item = {'type': item_elem.name} |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
318 item.update(item_elem.attr) |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
319 for child_elem in item_elem.children(): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
320 item[child_elem.name] = child_elem.gettext() |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
321 report.items.append(item) |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
322 report.insert(db=db) |
116 | 323 |
629
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
324 # Collect attachments from the request body |
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
325 for attach_elem in elem.children(Recipe.ATTACH): |
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
326 attach_elem = list(attach_elem.children('file'))[0] # One file only |
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
327 filename = attach_elem.attr.get('filename') |
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
328 resource_id = attach_elem.attr.get('resource') == 'config' \ |
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
329 and build.config or build.resource.id |
751
db5f7388535d
Merge [821,822,823,824,825,826] from trunk. Thanks Walter\!
hodgestar
parents:
716
diff
changeset
|
330 |
629
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
331 try: # Delete attachment if it already exists |
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
332 old_attach = Attachment(self.env, 'build', |
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
333 parent_id=resource_id, filename=filename) |
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
334 old_attach.delete() |
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
335 except ResourceNotFound: |
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
336 pass |
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
337 attachment = Attachment(self.env, 'build', parent_id=resource_id) |
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
338 attachment.description = attach_elem.attr.get('description') |
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
339 attachment.author = req.authname |
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
340 fileobj = StringIO(attach_elem.gettext().decode('base64')) |
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
341 attachment.insert(filename, fileobj, fileobj.len, db=db) |
f3bb52da9e3c
0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build.
osimons
parents:
611
diff
changeset
|
342 |
392 | 343 # If this was the last step in the recipe we mark the build as |
344 # completed | |
459 | 345 if last_step: |
392 | 346 self.log.info('Slave %s completed build %d ("%s" as of [%s])', |
347 build.slave, build.id, build.config, build.rev) | |
348 build.stopped = step.stopped | |
349 | |
350 # Determine overall outcome of the build by checking the outcome | |
351 # of the individual steps against the "onerror" specification of | |
352 # each step in the recipe | |
353 for num, recipe_step in enumerate(recipe): | |
354 step = BuildStep.fetch(self.env, build.id, recipe_step.id) | |
355 if step.status == BuildStep.FAILURE: | |
755 | 356 if recipe_step.onerror == 'fail' or \ |
357 recipe_step.onerror == 'continue': | |
392 | 358 build.status = Build.FAILURE |
359 break | |
360 else: | |
361 build.status = Build.SUCCESS | |
362 | |
363 build.update(db=db) | |
364 | |
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
197
diff
changeset
|
365 db.commit() |
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
197
diff
changeset
|
366 |
459 | 367 if last_step: |
392 | 368 for listener in BuildSystem(self.env).listeners: |
369 listener.build_completed(build) | |
253 | 370 |
392 | 371 body = 'Build step processed' |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
372 self._send_response(req, 201, body, { |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
373 'Content-Type': 'text/plain', |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
374 'Content-Length': str(len(body)), |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
375 'Location': req.abs_href.builds( |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
632
diff
changeset
|
376 build.id, 'steps', stepname)}) |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
377 |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
378 |
82
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
379 def _parse_iso_datetime(string): |
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
380 """Minimal parser for ISO date-time strings. |
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
381 |
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
382 Return the time as floating point number. Only handles UTC timestamps |
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
383 without time zone information.""" |
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
384 try: |
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
385 string = string.split('.', 1)[0] # strip out microseconds |
312 | 386 return calendar.timegm(time.strptime(string, '%Y-%m-%dT%H:%M:%S')) |
82
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
387 except ValueError, e: |
392 | 388 raise ValueError('Invalid ISO date/time %r' % string) |