Mercurial > bitten > bitten-test
annotate bitten/master.py @ 377:a465b5dbfecf
Tabs and line length.
author | cmlenz |
---|---|
date | Tue, 24 Jul 2007 14:37:04 +0000 |
parents | 446092a2d2fe |
children | 0df178e07fdb |
rev | line source |
---|---|
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
1 # -*- coding: iso8859-1 -*- |
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
2 # |
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
3 # Copyright (C) 2005 Christopher Lenz <cmlenz@gmx.de> |
163 | 4 # All rights reserved. |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
5 # |
163 | 6 # This software is licensed as described in the file COPYING, which |
7 # you should have received as part of this distribution. The terms | |
8 # are also available at http://bitten.cmlenz.net/wiki/License. | |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
9 |
313 | 10 """Build master implementation. |
11 | |
12 This module is runnable as a script to launch the build master. The build | |
13 master starts a single process that handles connections to any number of build | |
14 slaves. | |
15 """ | |
16 | |
312 | 17 import calendar |
95
1984b2e01998
Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents:
94
diff
changeset
|
18 from datetime import datetime, timedelta |
15
06207499c58c
* Use logging in the BEEP core as well as in the master and slave scripts. Closes #4.
cmlenz
parents:
14
diff
changeset
|
19 import logging |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
20 import os |
83
42970c14524a
Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents:
82
diff
changeset
|
21 try: |
42970c14524a
Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents:
82
diff
changeset
|
22 set |
42970c14524a
Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents:
82
diff
changeset
|
23 except NameError: |
42970c14524a
Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents:
82
diff
changeset
|
24 from sets import Set as set |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
25 import sys |
47
083e848088ee
* Improvements to the model classes, and a couple of unit tests.
cmlenz
parents:
45
diff
changeset
|
26 import time |
18
591a5a836ecc
* {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents:
15
diff
changeset
|
27 |
591a5a836ecc
* {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents:
15
diff
changeset
|
28 from trac.env import Environment |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
29 from bitten.model import BuildConfig, Build, BuildStep, BuildLog, Report |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
30 from bitten.queue import BuildQueue |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
31 from bitten.trac_ext.main import BuildSystem |
277 | 32 from bitten.util import beep, xmlio |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
33 |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
34 log = logging.getLogger('bitten.master') |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
35 |
95
1984b2e01998
Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents:
94
diff
changeset
|
36 DEFAULT_CHECK_INTERVAL = 120 # 2 minutes |
1984b2e01998
Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents:
94
diff
changeset
|
37 |
1984b2e01998
Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents:
94
diff
changeset
|
38 |
18
591a5a836ecc
* {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents:
15
diff
changeset
|
39 class Master(beep.Listener): |
313 | 40 """BEEP listener implementation for the build master.""" |
18
591a5a836ecc
* {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents:
15
diff
changeset
|
41 |
348 | 42 def __init__(self, envs, ip, port, build_all=False, |
43 adjust_timestamps=False, | |
95
1984b2e01998
Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents:
94
diff
changeset
|
44 check_interval=DEFAULT_CHECK_INTERVAL): |
18
591a5a836ecc
* {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents:
15
diff
changeset
|
45 beep.Listener.__init__(self, ip, port) |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
46 self.profiles[OrchestrationProfileHandler.URI] = \ |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
47 OrchestrationProfileHandler |
130
091ead7d2876
Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build.
cmlenz
parents:
127
diff
changeset
|
48 self.adjust_timestamps = adjust_timestamps |
95
1984b2e01998
Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents:
94
diff
changeset
|
49 self.check_interval = check_interval |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
50 self.handlers = {} # Map of connected slaves keyed by name |
47
083e848088ee
* Improvements to the model classes, and a couple of unit tests.
cmlenz
parents:
45
diff
changeset
|
51 |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
52 self.queues = [] |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
53 for env in envs: |
348 | 54 self.queues.append(BuildQueue(env, build_all=build_all)) |
47
083e848088ee
* Improvements to the model classes, and a couple of unit tests.
cmlenz
parents:
45
diff
changeset
|
55 |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
56 self.schedule(self.check_interval, self._enqueue_builds) |
47
083e848088ee
* Improvements to the model classes, and a couple of unit tests.
cmlenz
parents:
45
diff
changeset
|
57 |
083e848088ee
* Improvements to the model classes, and a couple of unit tests.
cmlenz
parents:
45
diff
changeset
|
58 def close(self): |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
59 for queue in self.queues: |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
60 queue.reset_orphaned_builds() |
47
083e848088ee
* Improvements to the model classes, and a couple of unit tests.
cmlenz
parents:
45
diff
changeset
|
61 beep.Listener.close(self) |
18
591a5a836ecc
* {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents:
15
diff
changeset
|
62 |
277 | 63 def _enqueue_builds(self): |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
64 self.schedule(self.check_interval, self._enqueue_builds) |
18
591a5a836ecc
* {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents:
15
diff
changeset
|
65 |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
66 for queue in self.queues: |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
67 queue.populate() |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
68 |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
69 self.schedule(self.check_interval * 0.2, self._initiate_builds) |
92
2e090827c63a
Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents:
90
diff
changeset
|
70 |
277 | 71 def _initiate_builds(self): |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
72 available_slaves = set([name for name in self.handlers |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
73 if not self.handlers[name].building]) |
228
a8c9dd7e3f71
* Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents:
227
diff
changeset
|
74 for idx, queue in enumerate(self.queues[:]): |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
75 build, slave = queue.get_next_pending_build(available_slaves) |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
76 if build: |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
77 self.handlers[slave].send_initiation(queue, build) |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
78 available_slaves.discard(slave) |
228
a8c9dd7e3f71
* Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents:
227
diff
changeset
|
79 self.queues.append(self.queues.pop(idx)) # Round robin |
18
591a5a836ecc
* {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents:
15
diff
changeset
|
80 |
83
42970c14524a
Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents:
82
diff
changeset
|
81 def register(self, handler): |
337
5c0625957185
Fix for #106. Thanks to Walter Bell for the report and patch.
cmlenz
parents:
313
diff
changeset
|
82 if handler.name in self.handlers: |
5c0625957185
Fix for #106. Thanks to Walter Bell for the report and patch.
cmlenz
parents:
313
diff
changeset
|
83 # The slave is for some reason still registered... this shouldn't |
5c0625957185
Fix for #106. Thanks to Walter Bell for the report and patch.
cmlenz
parents:
313
diff
changeset
|
84 # happen in theory, but apparently it does in the real world (see |
5c0625957185
Fix for #106. Thanks to Walter Bell for the report and patch.
cmlenz
parents:
313
diff
changeset
|
85 # #106). We simply unregister it before trying to register it |
5c0625957185
Fix for #106. Thanks to Walter Bell for the report and patch.
cmlenz
parents:
313
diff
changeset
|
86 # again. |
5c0625957185
Fix for #106. Thanks to Walter Bell for the report and patch.
cmlenz
parents:
313
diff
changeset
|
87 self.unregister(handler) |
5c0625957185
Fix for #106. Thanks to Walter Bell for the report and patch.
cmlenz
parents:
313
diff
changeset
|
88 |
83
42970c14524a
Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents:
82
diff
changeset
|
89 any_match = False |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
90 for queue in self.queues: |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
91 if queue.register_slave(handler.name, handler.info): |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
92 any_match = True |
83
42970c14524a
Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents:
82
diff
changeset
|
93 |
42970c14524a
Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents:
82
diff
changeset
|
94 if not any_match: |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
95 log.warning('Slave %s does not match any of the configured target ' |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
96 'platforms', handler.name) |
83
42970c14524a
Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents:
82
diff
changeset
|
97 return False |
42970c14524a
Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents:
82
diff
changeset
|
98 |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
99 self.handlers[handler.name] = handler |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
100 self.schedule(self.check_interval * 0.2, self._initiate_builds) |
95
1984b2e01998
Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents:
94
diff
changeset
|
101 |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
102 log.info('Registered slave "%s"', handler.name) |
83
42970c14524a
Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents:
82
diff
changeset
|
103 return True |
42970c14524a
Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents:
82
diff
changeset
|
104 |
42970c14524a
Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents:
82
diff
changeset
|
105 def unregister(self, handler): |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
106 if handler.name not in self.handlers: |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
107 return |
83
42970c14524a
Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents:
82
diff
changeset
|
108 |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
109 for queue in self.queues: |
253 | 110 if queue.unregister_slave(handler.name): |
111 for build in list(Build.select(queue.env, slave=handler.name, | |
112 status=Build.IN_PROGRESS)): | |
113 handler._build_aborted(queue, build) | |
114 | |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
115 del self.handlers[handler.name] |
184
fbf949f4c706
Allow invalidation of builds from the web interface. This results in the build being reset to ''PENDING'' status, and all build logs, slave information and reports deleted. Basically initiates a rebuild for a specific revision and target platform.
cmlenz
parents:
163
diff
changeset
|
116 |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
117 log.info('Unregistered slave "%s"', handler.name) |
83
42970c14524a
Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents:
82
diff
changeset
|
118 |
56 | 119 |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
120 class OrchestrationProfileHandler(beep.ProfileHandler): |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
121 """Handler for communication on the Bitten build orchestration profile from |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
122 the perspective of the build master. |
313 | 123 |
124 An instance of this class is associated with exactly one remote build slave. | |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
125 """ |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
126 URI = 'http://bitten.cmlenz.net/beep/orchestration' |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
127 |
51
5caccd7b247e
Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents:
49
diff
changeset
|
128 def handle_connect(self): |
18
591a5a836ecc
* {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents:
15
diff
changeset
|
129 self.master = self.session.listener |
591a5a836ecc
* {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents:
15
diff
changeset
|
130 assert self.master |
47
083e848088ee
* Improvements to the model classes, and a couple of unit tests.
cmlenz
parents:
45
diff
changeset
|
131 self.name = None |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
132 self.building = False |
82
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
133 self.info = {} |
24
cad6d28b8975
* Proper separation between {{{beep.ProfileHandler}}} instances between different channels.
cmlenz
parents:
19
diff
changeset
|
134 |
cad6d28b8975
* Proper separation between {{{beep.ProfileHandler}}} instances between different channels.
cmlenz
parents:
19
diff
changeset
|
135 def handle_disconnect(self): |
83
42970c14524a
Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents:
82
diff
changeset
|
136 self.master.unregister(self) |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
137 |
88 | 138 def handle_msg(self, msgno, payload): |
139 assert payload.content_type == beep.BEEP_XML | |
140 elem = xmlio.parse(payload.body) | |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
141 |
51
5caccd7b247e
Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents:
49
diff
changeset
|
142 if elem.name == 'register': |
69 | 143 self.name = elem.attr['name'] |
127
a9443f673344
Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29.
cmlenz
parents:
119
diff
changeset
|
144 self.info[Build.IP_ADDRESS] = self.session.addr[0] |
51
5caccd7b247e
Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents:
49
diff
changeset
|
145 for child in elem.children(): |
5caccd7b247e
Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents:
49
diff
changeset
|
146 if child.name == 'platform': |
82
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
147 self.info[Build.MACHINE] = child.gettext() |
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
148 self.info[Build.PROCESSOR] = child.attr.get('processor') |
51
5caccd7b247e
Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents:
49
diff
changeset
|
149 elif child.name == 'os': |
82
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
150 self.info[Build.OS_NAME] = child.gettext() |
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
151 self.info[Build.OS_FAMILY] = child.attr.get('family') |
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
152 self.info[Build.OS_VERSION] = child.attr.get('version') |
127
a9443f673344
Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29.
cmlenz
parents:
119
diff
changeset
|
153 elif child.name == 'package': |
a9443f673344
Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29.
cmlenz
parents:
119
diff
changeset
|
154 for name, value in child.attr.items(): |
a9443f673344
Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29.
cmlenz
parents:
119
diff
changeset
|
155 if name == 'name': |
a9443f673344
Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29.
cmlenz
parents:
119
diff
changeset
|
156 continue |
a9443f673344
Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29.
cmlenz
parents:
119
diff
changeset
|
157 self.info[child.attr['name'] + '.' + name] = value |
14
1733c601d2f8
Refactored the asyncore loop and shutdown procedure into {{{beep.Initiator}}}.
cmlenz
parents:
13
diff
changeset
|
158 |
83
42970c14524a
Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents:
82
diff
changeset
|
159 if not self.master.register(self): |
277 | 160 raise beep.ProtocolError(550, 'Nothing for you to build here, ' |
161 'please move along') | |
18
591a5a836ecc
* {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents:
15
diff
changeset
|
162 |
29 | 163 xml = xmlio.Element('ok') |
88 | 164 self.channel.send_rpy(msgno, beep.Payload(xml)) |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
165 |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
166 def send_initiation(self, queue, build): |
377 | 167 log.info('Initiating build of %d ("%s" as of [%s]) on slave %s', |
168 build.id, build.config, build.rev, self.name) | |
284
ddc93fe34cad
Fix a bug in the build master (introduced in [289]) where multiple slaves matching the same target platform would potentially start the same build. This is done by setting the build to `IN_PROGRESS` as soon as the build initiation is sent, as opposed to when the slave accepts the build request.
cmlenz
parents:
281
diff
changeset
|
169 |
ddc93fe34cad
Fix a bug in the build master (introduced in [289]) where multiple slaves matching the same target platform would potentially start the same build. This is done by setting the build to `IN_PROGRESS` as soon as the build initiation is sent, as opposed to when the slave accepts the build request.
cmlenz
parents:
281
diff
changeset
|
170 build.slave = self.name |
ddc93fe34cad
Fix a bug in the build master (introduced in [289]) where multiple slaves matching the same target platform would potentially start the same build. This is done by setting the build to `IN_PROGRESS` as soon as the build initiation is sent, as opposed to when the slave accepts the build request.
cmlenz
parents:
281
diff
changeset
|
171 build.slave_info.update(self.info) |
ddc93fe34cad
Fix a bug in the build master (introduced in [289]) where multiple slaves matching the same target platform would potentially start the same build. This is done by setting the build to `IN_PROGRESS` as soon as the build initiation is sent, as opposed to when the slave accepts the build request.
cmlenz
parents:
281
diff
changeset
|
172 build.status = Build.IN_PROGRESS |
ddc93fe34cad
Fix a bug in the build master (introduced in [289]) where multiple slaves matching the same target platform would potentially start the same build. This is done by setting the build to `IN_PROGRESS` as soon as the build initiation is sent, as opposed to when the slave accepts the build request.
cmlenz
parents:
281
diff
changeset
|
173 build.update() |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
174 self.building = True |
284
ddc93fe34cad
Fix a bug in the build master (introduced in [289]) where multiple slaves matching the same target platform would potentially start the same build. This is done by setting the build to `IN_PROGRESS` as soon as the build initiation is sent, as opposed to when the slave accepts the build request.
cmlenz
parents:
281
diff
changeset
|
175 |
277 | 176 config = BuildConfig.fetch(queue.env, build.config) |
42
efa525876b1e
Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents:
34
diff
changeset
|
177 |
88 | 178 def handle_reply(cmd, msgno, ansno, payload): |
42
efa525876b1e
Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents:
34
diff
changeset
|
179 if cmd == 'ERR': |
88 | 180 if payload.content_type == beep.BEEP_XML: |
181 elem = xmlio.parse(payload.body) | |
51
5caccd7b247e
Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents:
49
diff
changeset
|
182 if elem.name == 'error': |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
183 log.warning('Slave %s refused build request: %s (%d)', |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
184 self.name, elem.gettext(), |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
185 int(elem.attr['code'])) |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
186 self.building = False |
284
ddc93fe34cad
Fix a bug in the build master (introduced in [289]) where multiple slaves matching the same target platform would potentially start the same build. This is done by setting the build to `IN_PROGRESS` as soon as the build initiation is sent, as opposed to when the slave accepts the build request.
cmlenz
parents:
281
diff
changeset
|
187 self._build_aborted(queue, build) |
51
5caccd7b247e
Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents:
49
diff
changeset
|
188 return |
5caccd7b247e
Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents:
49
diff
changeset
|
189 |
88 | 190 elem = xmlio.parse(payload.body) |
277 | 191 if elem.name != 'proceed': |
192 raise beep.ProtocolError(500) | |
51
5caccd7b247e
Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents:
49
diff
changeset
|
193 |
277 | 194 snapshots = queue.snapshots[config.name] |
195 snapshot = snapshots.get(build.rev) | |
196 if not snapshot: | |
197 # Request a snapshot for this build, and schedule a poll | |
198 # function that kicks off the snapshot transmission once the | |
199 # archive has been completely built | |
200 worker = snapshots.create(build.rev) | |
201 def _check_snapshot(): | |
202 worker.join(.5) | |
203 if worker.isAlive(): | |
204 self.master.schedule(2, _check_snapshot) | |
205 else: | |
281
33625fa61d6c
* If a slave disconnects after the master has started to create a snapshot archive for it, just remain calm and keep the archive in place.
cmlenz
parents:
277
diff
changeset
|
206 if self.name not in self.master.handlers: |
33625fa61d6c
* If a slave disconnects after the master has started to create a snapshot archive for it, just remain calm and keep the archive in place.
cmlenz
parents:
277
diff
changeset
|
207 # The slave disconnected while we were building |
33625fa61d6c
* If a slave disconnects after the master has started to create a snapshot archive for it, just remain calm and keep the archive in place.
cmlenz
parents:
277
diff
changeset
|
208 # the archive |
33625fa61d6c
* If a slave disconnects after the master has started to create a snapshot archive for it, just remain calm and keep the archive in place.
cmlenz
parents:
277
diff
changeset
|
209 return |
277 | 210 snapshot = snapshots.get(build.rev) |
211 if snapshot is None: | |
212 log.error('Failed to create snapshot archive for ' | |
213 '%s@%s', config.path, build.rev) | |
214 return | |
215 self.send_snapshot(queue, build, snapshot) | |
216 _check_snapshot() | |
217 else: | |
218 self.send_snapshot(queue, build, snapshot) | |
219 | |
287
6abd43d0cd8a
The build slave now stores snapshot archives and the corresponding work directories in project folders of the main work folder, to keep build configurations from different projects that share the same name separate. This also requires transmitting the project name (simply the name of the environment directory) with the build initiation.
cmlenz
parents:
284
diff
changeset
|
220 xml = xmlio.parse(config.recipe) |
6abd43d0cd8a
The build slave now stores snapshot archives and the corresponding work directories in project folders of the main work folder, to keep build configurations from different projects that share the same name separate. This also requires transmitting the project name (simply the name of the environment directory) with the build initiation.
cmlenz
parents:
284
diff
changeset
|
221 xml.attr['project'] = os.path.basename(queue.env.path) |
373
b81c67778df7
Flush pipes to the master before after each step so it shows up in the UI immediately, not just at the end of the build. Closes #18, #96
wbell
parents:
368
diff
changeset
|
222 self.channel.send_msg(beep.Payload(xml), |
b81c67778df7
Flush pipes to the master before after each step so it shows up in the UI immediately, not just at the end of the build. Closes #18, #96
wbell
parents:
368
diff
changeset
|
223 handle_reply=handle_reply, force_flush=True) |
51
5caccd7b247e
Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents:
49
diff
changeset
|
224 |
277 | 225 def send_snapshot(self, queue, build, snapshot): |
130
091ead7d2876
Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build.
cmlenz
parents:
127
diff
changeset
|
226 timestamp_delta = 0 |
091ead7d2876
Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build.
cmlenz
parents:
127
diff
changeset
|
227 if self.master.adjust_timestamps: |
091ead7d2876
Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build.
cmlenz
parents:
127
diff
changeset
|
228 d = datetime.now() - timedelta(seconds=self.master.check_interval) \ |
091ead7d2876
Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build.
cmlenz
parents:
127
diff
changeset
|
229 - datetime.fromtimestamp(build.rev_time) |
277 | 230 log.info('Warping timestamps by %s', d) |
130
091ead7d2876
Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build.
cmlenz
parents:
127
diff
changeset
|
231 timestamp_delta = d.days * 86400 + d.seconds |
091ead7d2876
Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build.
cmlenz
parents:
127
diff
changeset
|
232 |
88 | 233 def handle_reply(cmd, msgno, ansno, payload): |
51
5caccd7b247e
Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents:
49
diff
changeset
|
234 if cmd == 'ERR': |
277 | 235 if payload.content_type != beep.BEEP_XML: |
236 raise beep.ProtocolError(500) | |
88 | 237 elem = xmlio.parse(payload.body) |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
238 if elem.name == 'error': |
185
2c24d9a950ed
Add a `--dry-run` option to the build slave. This will result in the slave being registered and executing builds, but without submitting the progress and results of the build back to the server. Useful for getting the configuration of new slaves right without polluting the database with invalid builds.
cmlenz
parents:
184
diff
changeset
|
239 log.warning('Slave %s refused to start build: %s (%d)', |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
240 self.name, elem.gettext(), |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
241 int(elem.attr['code'])) |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
242 self.building = False |
284
ddc93fe34cad
Fix a bug in the build master (introduced in [289]) where multiple slaves matching the same target platform would potentially start the same build. This is done by setting the build to `IN_PROGRESS` as soon as the build initiation is sent, as opposed to when the slave accepts the build request.
cmlenz
parents:
281
diff
changeset
|
243 self._build_aborted(queue, build) |
80
dc1c7fc9b915
Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents:
76
diff
changeset
|
244 |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
245 elif cmd == 'ANS': |
277 | 246 if payload.content_type != beep.BEEP_XML: |
247 raise beep.ProtocolError(500) | |
88 | 248 elem = xmlio.parse(payload.body) |
374
446092a2d2fe
Don't accept build results from a slave if the build has been invalidated or it's being built by another slave. Closes #100.
wbell
parents:
373
diff
changeset
|
249 |
377 | 250 # verify that the build hasn't been modified out from |
251 # under us-- we don't want to accept any build information | |
252 # from builds that have been invalidated or claimed by | |
253 # other slaves. | |
254 current_build = Build.fetch(queue.env, build.id) | |
255 if current_build.status != Build.IN_PROGRESS or \ | |
256 current_build.slave != self.name: | |
257 raise beep.ProtocolError(550, 'Build %s has been ' | |
258 'invalidated, will not accept completed steps from ' | |
259 'slave %s' % (build.id, self.name)) | |
260 | |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
261 if elem.name == 'started': |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
262 self._build_started(queue, build, elem, timestamp_delta) |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
263 elif elem.name == 'step': |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
264 self._build_step_completed(queue, build, elem, |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
265 timestamp_delta) |
82
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
266 elif elem.name == 'completed': |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
267 self._build_completed(queue, build, elem, timestamp_delta) |
70 | 268 elif elem.name == 'aborted': |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
269 self._build_aborted(queue, build) |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
270 elif elem.name == 'error': |
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
271 build.status = Build.FAILURE |
80
dc1c7fc9b915
Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents:
76
diff
changeset
|
272 |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
273 elif cmd == 'NUL': |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
274 self.building = False |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
275 |
277 | 276 snapshot_name = os.path.basename(snapshot) |
277 message = beep.Payload(file(snapshot, 'rb'), | |
310
0ffdab3a7032
Transmit source archives as `tar.bz2` instead of as zip. See #76.
cmlenz
parents:
301
diff
changeset
|
278 content_type='application/tar', |
0ffdab3a7032
Transmit source archives as `tar.bz2` instead of as zip. See #76.
cmlenz
parents:
301
diff
changeset
|
279 content_encoding='bzip2', |
277 | 280 content_disposition=snapshot_name) |
42
efa525876b1e
Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents:
34
diff
changeset
|
281 self.channel.send_msg(message, handle_reply=handle_reply) |
efa525876b1e
Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents:
34
diff
changeset
|
282 |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
283 def _build_started(self, queue, build, elem, timestamp_delta=None): |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
284 build.started = int(_parse_iso_datetime(elem.attr['time'])) |
130
091ead7d2876
Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build.
cmlenz
parents:
127
diff
changeset
|
285 if timestamp_delta: |
091ead7d2876
Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build.
cmlenz
parents:
127
diff
changeset
|
286 build.started -= timestamp_delta |
253 | 287 build.update() |
288 | |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
289 log.info('Slave %s started build %d ("%s" as of [%s])', |
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
290 self.name, build.id, build.config, build.rev) |
253 | 291 for listener in BuildSystem(queue.env).listeners: |
292 listener.build_started(build) | |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
293 |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
294 def _build_step_completed(self, queue, build, elem, timestamp_delta=None): |
204
e1a53e70b43f
* Make deletion of build steps compatible with PySQLite2: records cannot be deleted inside a `select()` loop.
cmlenz
parents:
203
diff
changeset
|
295 log.debug('Slave %s completed step "%s" with status %s', self.name, |
e1a53e70b43f
* Make deletion of build steps compatible with PySQLite2: records cannot be deleted inside a `select()` loop.
cmlenz
parents:
203
diff
changeset
|
296 elem.attr['id'], elem.attr['result']) |
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
197
diff
changeset
|
297 |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
298 db = queue.env.get_db_cnx() |
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
197
diff
changeset
|
299 |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
300 step = BuildStep(queue.env, build=build.id, name=elem.attr['id'], |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
301 description=elem.attr.get('description')) |
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
302 step.started = int(_parse_iso_datetime(elem.attr['time'])) |
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
303 step.stopped = step.started + int(elem.attr['duration']) |
130
091ead7d2876
Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build.
cmlenz
parents:
127
diff
changeset
|
304 if timestamp_delta: |
091ead7d2876
Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build.
cmlenz
parents:
127
diff
changeset
|
305 step.started -= timestamp_delta |
091ead7d2876
Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build.
cmlenz
parents:
127
diff
changeset
|
306 step.stopped -= timestamp_delta |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
307 if elem.attr['result'] == 'failure': |
377 | 308 log.warning('Build %s Step %s failed', build.id, elem.attr['id']) |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
309 step.status = BuildStep.FAILURE |
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
310 else: |
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
311 step.status = BuildStep.SUCCESS |
277 | 312 step.errors += [error.gettext() for error in elem.children('error')] |
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
110
diff
changeset
|
313 step.insert(db=db) |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
314 |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
315 for idx, log_elem in enumerate(elem.children('log')): |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
316 build_log = BuildLog(queue.env, build=build.id, step=step.name, |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
317 generator=log_elem.attr.get('generator'), |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
318 orderno=idx) |
115
16d69eb6e047
Add support for XML fragments to the {{{xmlio}}} module, so that build output and reports don't need to be nested in a meaningless element (such as {{{<log type="distutils"><messages><message ...>}}}).
cmlenz
parents:
112
diff
changeset
|
319 for message_elem in log_elem.children('message'): |
16d69eb6e047
Add support for XML fragments to the {{{xmlio}}} module, so that build output and reports don't need to be nested in a meaningless element (such as {{{<log type="distutils"><messages><message ...>}}}).
cmlenz
parents:
112
diff
changeset
|
320 build_log.messages.append((message_elem.attr['level'], |
16d69eb6e047
Add support for XML fragments to the {{{xmlio}}} module, so that build output and reports don't need to be nested in a meaningless element (such as {{{<log type="distutils"><messages><message ...>}}}).
cmlenz
parents:
112
diff
changeset
|
321 message_elem.gettext())) |
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
110
diff
changeset
|
322 build_log.insert(db=db) |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
323 |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
324 for report_elem in elem.children('report'): |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
325 report = Report(queue.env, build=build.id, step=step.name, |
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
204
diff
changeset
|
326 category=report_elem.attr.get('category'), |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
204
diff
changeset
|
327 generator=report_elem.attr.get('generator')) |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
328 for item_elem in report_elem.children(): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
329 item = {'type': item_elem.name} |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
330 item.update(item_elem.attr) |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
331 for child_elem in item_elem.children(): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
332 item[child_elem.name] = child_elem.gettext() |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
333 report.items.append(item) |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
202
diff
changeset
|
334 report.insert(db=db) |
116 | 335 |
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
197
diff
changeset
|
336 db.commit() |
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
197
diff
changeset
|
337 |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
338 def _build_completed(self, queue, build, elem, timestamp_delta=None): |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
339 build.stopped = int(_parse_iso_datetime(elem.attr['time'])) |
130
091ead7d2876
Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build.
cmlenz
parents:
127
diff
changeset
|
340 if timestamp_delta: |
091ead7d2876
Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build.
cmlenz
parents:
127
diff
changeset
|
341 build.stopped -= timestamp_delta |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
342 if elem.attr['result'] == 'failure': |
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
343 build.status = Build.FAILURE |
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
344 else: |
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
345 build.status = Build.SUCCESS |
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
197
diff
changeset
|
346 build.update() |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
347 |
253 | 348 log.info('Slave %s completed build %d ("%s" as of [%s]) with status %s', |
349 self.name, build.id, build.config, build.rev, | |
350 build.status == Build.FAILURE and 'FAILURE' or 'SUCCESS') | |
351 for listener in BuildSystem(queue.env).listeners: | |
352 listener.build_completed(build) | |
353 | |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
354 def _build_aborted(self, queue, build): |
204
e1a53e70b43f
* Make deletion of build steps compatible with PySQLite2: records cannot be deleted inside a `select()` loop.
cmlenz
parents:
203
diff
changeset
|
355 log.info('Slave %s aborted build %d ("%s" as of [%s])', |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
356 self.name, build.id, build.config, build.rev) |
253 | 357 for listener in BuildSystem(queue.env).listeners: |
358 listener.build_aborted(build) | |
184
fbf949f4c706
Allow invalidation of builds from the web interface. This results in the build being reset to ''PENDING'' status, and all build logs, slave information and reports deleted. Basically initiates a rebuild for a specific revision and target platform.
cmlenz
parents:
163
diff
changeset
|
359 |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
360 db = queue.env.get_db_cnx() |
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
197
diff
changeset
|
361 |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
362 for step in list(BuildStep.select(queue.env, build=build.id, db=db)): |
184
fbf949f4c706
Allow invalidation of builds from the web interface. This results in the build being reset to ''PENDING'' status, and all build logs, slave information and reports deleted. Basically initiates a rebuild for a specific revision and target platform.
cmlenz
parents:
163
diff
changeset
|
363 step.delete(db=db) |
fbf949f4c706
Allow invalidation of builds from the web interface. This results in the build being reset to ''PENDING'' status, and all build logs, slave information and reports deleted. Basically initiates a rebuild for a specific revision and target platform.
cmlenz
parents:
163
diff
changeset
|
364 |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
365 build.slave = None |
253 | 366 build.slave_info = {} |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
367 build.started = 0 |
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
368 build.status = Build.PENDING |
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
197
diff
changeset
|
369 build.update(db=db) |
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
197
diff
changeset
|
370 |
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
197
diff
changeset
|
371 db.commit() |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
96
diff
changeset
|
372 |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
373 |
82
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
374 def _parse_iso_datetime(string): |
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
375 """Minimal parser for ISO date-time strings. |
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
376 |
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
377 Return the time as floating point number. Only handles UTC timestamps |
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
378 without time zone information.""" |
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
379 try: |
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
380 string = string.split('.', 1)[0] # strip out microseconds |
312 | 381 return calendar.timegm(time.strptime(string, '%Y-%m-%dT%H:%M:%S')) |
82
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
382 except ValueError, e: |
368 | 383 raise ValueError('Invalid ISO date/time %s (%s)' % (string, e)) |
82
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
384 |
31 | 385 def main(): |
313 | 386 """Main entry point for running the build master.""" |
56 | 387 from bitten import __version__ as VERSION |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
388 from optparse import OptionParser |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
389 |
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
110
diff
changeset
|
390 # Parse command-line arguments |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
391 parser = OptionParser(usage='usage: %prog [options] ENV_PATHS', |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
392 version='%%prog %s' % VERSION) |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
393 parser.add_option('-p', '--port', action='store', type='int', dest='port', |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
394 help='port number to use') |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
395 parser.add_option('-H', '--host', action='store', dest='host', |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
396 metavar='HOSTNAME', |
33
d8d44216258a
Exit the slave script when the master disconnects; and other minor fixes.
cmlenz
parents:
31
diff
changeset
|
397 help='the host name or IP address to bind to') |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
398 parser.add_option('-l', '--log', dest='logfile', metavar='FILENAME', |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
399 help='write log messages to FILENAME') |
95
1984b2e01998
Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents:
94
diff
changeset
|
400 parser.add_option('-i', '--interval', dest='interval', metavar='SECONDS', |
1984b2e01998
Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents:
94
diff
changeset
|
401 default=DEFAULT_CHECK_INTERVAL, type='int', |
1984b2e01998
Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents:
94
diff
changeset
|
402 help='poll interval for changeset detection') |
348 | 403 parser.add_option('--build-all', action='store_true', dest='buildall', |
404 help='build older revisions even when a build for a ' | |
405 'newer revision has already been performed') | |
242
372d1de2e3ec
* Fixes to the `<c:configure>` command added in [247]: Set current directory when invoking the script, and correctly pass `CFLAGS` and `CXXFLAGS`.
cmlenz
parents:
228
diff
changeset
|
406 parser.add_option('--timewarp', action='store_true', dest='timewarp', |
267 | 407 help='adjust timestamps of builds to be near the ' |
130
091ead7d2876
Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build.
cmlenz
parents:
127
diff
changeset
|
408 'timestamps of the corresponding changesets') |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
409 parser.add_option('--debug', action='store_const', dest='loglevel', |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
410 const=logging.DEBUG, help='enable debugging output') |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
411 parser.add_option('-v', '--verbose', action='store_const', dest='loglevel', |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
412 const=logging.INFO, help='print as much as possible') |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
413 parser.add_option('-q', '--quiet', action='store_const', dest='loglevel', |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
414 const=logging.ERROR, help='print as little as possible') |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
415 parser.set_defaults(port=7633, loglevel=logging.WARNING) |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
416 options, args = parser.parse_args() |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
417 |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
418 if len(args) < 1: |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
419 parser.error('incorrect number of arguments') |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
420 |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
421 # Configure logging |
157
2efdc69e63c3
Some style/documentation improvements to make Pylint happier.
cmlenz
parents:
152
diff
changeset
|
422 logger = logging.getLogger('bitten') |
2efdc69e63c3
Some style/documentation improvements to make Pylint happier.
cmlenz
parents:
152
diff
changeset
|
423 logger.setLevel(options.loglevel) |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
424 handler = logging.StreamHandler() |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
425 if options.logfile: |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
426 handler.setLevel(logging.WARNING) |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
427 else: |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
428 handler.setLevel(options.loglevel) |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
429 formatter = logging.Formatter('%(message)s') |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
430 handler.setFormatter(formatter) |
157
2efdc69e63c3
Some style/documentation improvements to make Pylint happier.
cmlenz
parents:
152
diff
changeset
|
431 logger.addHandler(handler) |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
432 if options.logfile: |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
433 handler = logging.FileHandler(options.logfile) |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
434 handler.setLevel(options.loglevel) |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
435 formatter = logging.Formatter('%(asctime)s [%(name)s] %(levelname)s: ' |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
436 '%(message)s') |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
437 handler.setFormatter(formatter) |
157
2efdc69e63c3
Some style/documentation improvements to make Pylint happier.
cmlenz
parents:
152
diff
changeset
|
438 logger.addHandler(handler) |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
439 |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
440 port = options.port |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
441 if not (1 <= port <= 65535): |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
442 parser.error('port must be an integer in the range 1-65535') |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
443 |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
444 host = options.host |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
445 if not host: |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
446 import socket |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
447 ip = socket.gethostbyname(socket.gethostname()) |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
448 try: |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
449 host = socket.gethostbyaddr(ip)[0] |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
450 except socket.error, e: |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
92
diff
changeset
|
451 log.warning('Reverse host name lookup failed (%s)', e) |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
452 host = ip |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
453 |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
454 envs = [] |
301
d486e34084af
Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project.
cmlenz
parents:
287
diff
changeset
|
455 env_names = set() |
d486e34084af
Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project.
cmlenz
parents:
287
diff
changeset
|
456 for env_path in [os.path.normpath(arg) for arg in args]: |
d486e34084af
Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project.
cmlenz
parents:
287
diff
changeset
|
457 if not os.path.isdir(env_path): |
d486e34084af
Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project.
cmlenz
parents:
287
diff
changeset
|
458 log.warning('Ignoring %s: not a directory', env_path) |
228
a8c9dd7e3f71
* Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents:
227
diff
changeset
|
459 continue |
301
d486e34084af
Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project.
cmlenz
parents:
287
diff
changeset
|
460 env_name = os.path.basename(env_path) |
d486e34084af
Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project.
cmlenz
parents:
287
diff
changeset
|
461 if env_name in env_names: |
d486e34084af
Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project.
cmlenz
parents:
287
diff
changeset
|
462 log.warning('Ignoring %s: duplicate project name "%s"', env_path, |
d486e34084af
Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project.
cmlenz
parents:
287
diff
changeset
|
463 env_name) |
287
6abd43d0cd8a
The build slave now stores snapshot archives and the corresponding work directories in project folders of the main work folder, to keep build configurations from different projects that share the same name separate. This also requires transmitting the project name (simply the name of the environment directory) with the build initiation.
cmlenz
parents:
284
diff
changeset
|
464 continue |
301
d486e34084af
Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project.
cmlenz
parents:
287
diff
changeset
|
465 env_names.add(env_name) |
d486e34084af
Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project.
cmlenz
parents:
287
diff
changeset
|
466 env = Environment(env_path) |
227
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
467 if BuildSystem(env): |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
468 if env.needs_upgrade(): |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
469 log.warning('Environment at %s needs to be upgraded', env.path) |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
470 continue |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
471 envs.append(env) |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
472 if not envs: |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
473 log.error('None of the specified environments has support for Bitten') |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
474 sys.exit(2) |
014bc6c29dff
* Factor build queue logic into a class separate from the build master.
cmlenz
parents:
213
diff
changeset
|
475 |
348 | 476 master = Master(envs, host, port, build_all=options.buildall, |
477 adjust_timestamps=options.timewarp, | |
130
091ead7d2876
Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build.
cmlenz
parents:
127
diff
changeset
|
478 check_interval=options.interval) |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
479 try: |
42
efa525876b1e
Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents:
34
diff
changeset
|
480 master.run(timeout=5.0) |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
481 except KeyboardInterrupt: |
34
6da9468a6879
The build master now gracefully exits by first terminating all active sessions. Fixes #7.
cmlenz
parents:
33
diff
changeset
|
482 master.quit() |
31 | 483 |
484 if __name__ == '__main__': | |
33
d8d44216258a
Exit the slave script when the master disconnects; and other minor fixes.
cmlenz
parents:
31
diff
changeset
|
485 main() |