Mercurial > bitten > bitten-test

annotate bitten/master.py @ 88:8ae753a22494

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Some refactoring of the BEEP implementation: * Renamed {{{MIMEMessage}}} to {{{Payload}}} * No longer is a subclass of {{{email.Message.Message}}}
author cmlenz
date Thu, 14 Jul 2005 14:20:42 +0000
parents 110bfa3cbc32
children 2c4e104afef8
rev   line source
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
1 # -*- coding: iso8859-1 -*-
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
2 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
3 # Copyright (C) 2005 Christopher Lenz <cmlenz@gmx.de>
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
4 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
5 # Bitten is free software; you can redistribute it and/or
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
6 # modify it under the terms of the GNU General Public License as
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
7 # published by the Free Software Foundation; either version 2 of the
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
8 # License, or (at your option) any later version.
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
9 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
10 # Trac is distributed in the hope that it will be useful,
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
11 # but WITHOUT ANY WARRANTY; without even the implied warranty of
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
12 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
13 # General Public License for more details.
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
14 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
15 # You should have received a copy of the GNU General Public License
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
16 # along with this program; if not, write to the Free Software
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
17 # Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
18 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
19 # Author: Christopher Lenz <cmlenz@gmx.de>
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
20
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
21 from itertools import ifilter
15
06207499c58c * Use logging in the BEEP core as well as in the master and slave scripts. Closes #4.
cmlenz
parents: 14
diff changeset
22 import logging
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
23 import os.path
76
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
24 import re
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
25 try:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
26 set
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
27 except NameError:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
28 from sets import Set as set
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
29 import time
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
30
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
31 from trac.env import Environment
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
32 from bitten.model import BuildConfig, TargetPlatform, Build, BuildStep
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
33 from bitten.util import archive, beep, xmlio
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
34
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
35
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
36 class Master(beep.Listener):
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
37
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
38 TRIGGER_INTERVAL = 10
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
39
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
40 def __init__(self, env_path, ip, port):
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
41 beep.Listener.__init__(self, ip, port)
24
cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels.
cmlenz
parents: 19
diff changeset
42 self.profiles[OrchestrationProfileHandler.URI] = OrchestrationProfileHandler
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
43 self.env = Environment(env_path)
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
44
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
45 self.slaves = {}
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
46
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
47 # path to generated snapshot archives, key is (config name, revision)
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
48 self.snapshots = {}
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
49
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
50 self.schedule(self.TRIGGER_INTERVAL, self._check_build_triggers)
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
51
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
52 def close(self):
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
53 # Remove all pending builds
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
54 for build in Build.select(self.env, status=Build.PENDING):
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
55 build.delete()
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
56 beep.Listener.close(self)
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
57
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
58 def _check_build_triggers(self, master, when):
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
59 self.schedule(self.TRIGGER_INTERVAL, self._check_build_triggers)
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
60
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
61 repos = self.env.get_repository()
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
62 try:
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
63 repos.sync()
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
64
45
80bc0fae3ed1 Renamed {{{Configuration}}} to {{{BuildConfig}}}.
cmlenz
parents: 42
diff changeset
65 for config in BuildConfig.select(self.env):
76
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
66 logging.debug('Checking for changes to "%s" at %s',
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
67 config.label, config.path)
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
68 node = repos.get_node(config.path)
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
69 for path, rev, chg in node.get_history():
76
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
70 enqueued = False
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
71 for platform in TargetPlatform.select(self.env, config.name):
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
72 # Check whether the latest revision of the configuration
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
73 # has already been built on this platform
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
74 builds = Build.select(self.env, config.name, rev,
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
75 platform.id)
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
76 if not list(builds):
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
77 logging.info('Enqueuing build of configuration "%s"'
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
78 ' at revision [%s] on %s', config.name,
76
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
79 rev, platform.name)
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
80 build = Build(self.env)
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
81 build.config = config.name
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
82 build.rev = rev
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
83 build.rev_time = repos.get_changeset(rev).date
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
84 build.platform = platform.id
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
85 build.insert()
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
86 enqueued = True
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
87 if enqueued:
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
88 break
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
89 finally:
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
90 repos.close()
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
91
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
92 self.schedule(5, self._check_build_queue)
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
93
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
94 def _check_build_queue(self, master, when):
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
95 if not self.slaves:
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
96 return
49
b4c51e32952b Build slave now also handles snapshot archives in bzip2 and zip format.
cmlenz
parents: 48
diff changeset
97 logging.debug('Checking for pending builds...')
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
98 for build in Build.select(self.env, status=Build.PENDING):
86
110bfa3cbc32 Slaves were getting associated with the wrong target platform.
cmlenz
parents: 85
diff changeset
99 for slave in self.slaves.get(build.platform, []):
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
100 active_builds = Build.select(self.env, slave=slave.name,
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
101 status=Build.IN_PROGRESS)
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
102 if not list(active_builds):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
103 slave.send_initiation(build)
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
104 return
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
105
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
106 def get_snapshot(self, build, type, encoding):
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
107 formats = {
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
108 ('application/tar', 'bzip2'): 'bzip2',
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
109 ('application/tar', 'gzip'): 'bzip',
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
110 ('application/tar', None): 'tar',
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
111 ('application/zip', None): 'zip',
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
112 }
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
113 if not (build.config, build.rev, type, encoding) in self.snapshots:
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
114 config = BuildConfig(self.env, build.config)
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
115 snapshot = archive.pack(self.env, path=config.path, rev=build.rev,
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
116 prefix=config.name,
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
117 format=formats[(type, encoding)])
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
118 logging.info('Prepared snapshot archive at %s' % snapshot)
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
119 self.snapshots[(build.config, build.rev, type, encoding)] = snapshot
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
120 return self.snapshots[(build.config, build.rev, type, encoding)]
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
121
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
122 def register(self, handler):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
123 any_match = False
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
124 for config in BuildConfig.select(self.env):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
125 for platform in TargetPlatform.select(self.env, config=config.name):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
126 if not platform.id in self.slaves:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
127 self.slaves[platform.id] = set()
85
acb7b67b8152 Fix matching of slave properties against target platform rules.
cmlenz
parents: 84
diff changeset
128 match = True
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
129 for property, pattern in ifilter(None, platform.rules):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
130 try:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
131 if not re.match(pattern, handler.info.get(property)):
86
110bfa3cbc32 Slaves were getting associated with the wrong target platform.
cmlenz
parents: 85
diff changeset
132 match = False
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
133 break
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
134 except re.error, e:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
135 logging.error('Invalid platform matching pattern "%s"',
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
136 pattern, exc_info=True)
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
137 match = False
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
138 break
84
30d09249de80 Fix var name in slave registration.
cmlenz
parents: 83
diff changeset
139 if match:
86
110bfa3cbc32 Slaves were getting associated with the wrong target platform.
cmlenz
parents: 85
diff changeset
140 logging.info('Slave %s matched target platform %s',
110bfa3cbc32 Slaves were getting associated with the wrong target platform.
cmlenz
parents: 85
diff changeset
141 handler.name, platform.name)
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
142 self.slaves[platform.id].add(handler)
86
110bfa3cbc32 Slaves were getting associated with the wrong target platform.
cmlenz
parents: 85
diff changeset
143 any_match = True
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
144
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
145 if not any_match:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
146 logging.warning('Slave %s does not match any of the configured '
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
147 'target platforms', handler.name)
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
148 return False
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
149
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
150 logging.info('Registered slave "%s"', handler.name)
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
151 return True
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
152
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
153 def unregister(self, handler):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
154 for slaves in self.slaves.values():
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
155 slaves.discard(handler)
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
156
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
157 for build in Build.select(self.env, slave=handler.name,
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
158 status=Build.IN_PROGRESS):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
159 logging.info('Build [%s] of "%s" by %s cancelled', build.rev,
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
160 build.config, handler.name)
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
161 build.slave = None
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
162 build.status = Build.PENDING
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
163 build.started = 0
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
164 build.update()
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
165 break
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
166 logging.info('Unregistered slave "%s"', handler.name)
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
167
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
168
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
169 class OrchestrationProfileHandler(beep.ProfileHandler):
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
170 """Handler for communication on the Bitten build orchestration profile from
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
171 the perspective of the build master.
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
172 """
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
173 URI = 'http://bitten.cmlenz.net/beep/orchestration'
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
174
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
175 def handle_connect(self):
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
176 self.master = self.session.listener
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
177 assert self.master
69
b92d7c7d70fd Record build slave properties in database.
cmlenz
parents: 66
diff changeset
178 self.env = self.master.env
b92d7c7d70fd Record build slave properties in database.
cmlenz
parents: 66
diff changeset
179 assert self.env
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
180 self.name = None
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
181 self.info = {}
24
cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels.
cmlenz
parents: 19
diff changeset
182
cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels.
cmlenz
parents: 19
diff changeset
183 def handle_disconnect(self):
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
184 self.master.unregister(self)
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
185
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
186 def handle_msg(self, msgno, payload):
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
187 assert payload.content_type == beep.BEEP_XML
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
188 elem = xmlio.parse(payload.body)
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
189
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
190 if elem.name == 'register':
69
b92d7c7d70fd Record build slave properties in database.
cmlenz
parents: 66
diff changeset
191 self.name = elem.attr['name']
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
192 for child in elem.children():
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
193 if child.name == 'platform':
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
194 self.info[Build.MACHINE] = child.gettext()
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
195 self.info[Build.PROCESSOR] = child.attr.get('processor')
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
196 elif child.name == 'os':
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
197 self.info[Build.OS_NAME] = child.gettext()
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
198 self.info[Build.OS_FAMILY] = child.attr.get('family')
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
199 self.info[Build.OS_VERSION] = child.attr.get('version')
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
200 self.info[Build.IP_ADDRESS] = self.session.addr[0]
14
1733c601d2f8 Refactored the asyncore loop and shutdown procedure into {{{beep.Initiator}}}.
cmlenz
parents: 13
diff changeset
201
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
202 if not self.master.register(self):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
203 xml = xmlio.Element('error', code=550)[
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
204 'Nothing for you to build here, please move along'
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
205 ]
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
206 self.channel.send_err(msgno, beep.Payload(xml))
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
207 return
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
208
29
2adc3480e4aa Some cleanup and additional docstrings.
cmlenz
parents: 24
diff changeset
209 xml = xmlio.Element('ok')
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
210 self.channel.send_rpy(msgno, beep.Payload(xml))
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
211
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
212 def send_initiation(self, build):
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
213 logging.debug('Initiating build of "%s" on slave %s', build.config,
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
214 self.name)
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
215
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
216 def handle_reply(cmd, msgno, ansno, payload):
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
217 if cmd == 'ERR':
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
218 if payload.content_type == beep.BEEP_XML:
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
219 elem = xmlio.parse(payload.body)
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
220 if elem.name == 'error':
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
221 logging.warning('Slave %s refused build request: '
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
222 '%s (%d)', self.name, elem.gettext(),
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
223 int(elem.attr['code']))
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
224 return
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
225
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
226 elem = xmlio.parse(payload.body)
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
227 assert elem.name == 'proceed'
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
228 type = encoding = None
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
229 for child in elem.children('accept'):
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
230 type, encoding = child.attr['type'], child.attr.get('encoding')
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
231 if (type, encoding) in (('application/tar', 'gzip'),
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
232 ('application/tar', 'bzip2'),
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
233 ('application/tar', None),
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
234 ('application/zip', None)):
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
235 break
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
236 type = None
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
237 if not type:
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
238 xml = xmlio.Element('error', code=550)[
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
239 'None of the accepted archive formats supported'
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
240 ]
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
241 self.channel.send_err(beep.Payload(xml))
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
242 return
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
243 self.send_snapshot(build, type, encoding)
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
244
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
245 xml = xmlio.Element('build', recipe='recipe.xml')
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
246 self.channel.send_msg(beep.Payload(xml), handle_reply=handle_reply)
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
247
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
248 def send_snapshot(self, build, type, encoding):
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
249
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
250 def handle_reply(cmd, msgno, ansno, payload):
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
251 if cmd == 'ERR':
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
252 assert payload.content_type == beep.BEEP_XML
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
253 elem = xmlio.parse(payload.body)
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
254 if elem.name == 'error':
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
255 logging.warning('Slave %s did not accept archive: %s (%d)',
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
256 self.name, elem.gettext(),
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
257 int(elem.attr['code']))
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
258
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
259 if cmd == 'ANS':
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
260 elem = xmlio.parse(payload.body)
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
261
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
262 if elem.name == 'started':
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
263 build.slave = self.name
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
264 build.slave_info.update(self.info)
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
265 build.started = int(_parse_iso_datetime(elem.attr['time']))
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
266 build.status = Build.IN_PROGRESS
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
267 build.update()
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
268 logging.info('Slave %s started build of "%s" as of [%s]',
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
269 self.name, build.config, build.rev)
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
270
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
271 elif elem.name == 'step':
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
272 logging.info('Slave completed step "%s"', elem.attr['id'])
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
273 step = BuildStep(self.env)
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
274 step.build = build.id
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
275 step.name = elem.attr['id']
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
276 step.description = elem.attr.get('description')
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
277 step.started = int(_parse_iso_datetime(elem.attr['time']))
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
278 step.stopped = step.started + int(elem.attr['duration'])
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
279 step.log = elem.gettext().strip()
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
280 if elem.attr['result'] == 'failure':
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
281 logging.warning('Step failed: %s', elem.gettext())
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
282 step.status = BuildStep.FAILURE
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
283 else:
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
284 step.status = BuildStep.SUCCESS
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
285 step.insert()
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
286
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
287 elif elem.name == 'completed':
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
288 logging.info('Slave %s completed build of "%s" as of [%s]',
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
289 self.name, build.config, build.rev)
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
290 build.stopped = int(_parse_iso_datetime(elem.attr['time']))
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
291 if elem.attr['result'] == 'failure':
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
292 build.status = Build.FAILURE
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
293 else:
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
294 build.status = Build.SUCCESS
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
295
70
ccd03b6f04ef Updated DTD for orchestration profile.
cmlenz
parents: 69
diff changeset
296 elif elem.name == 'aborted':
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
297 logging.info('Slave "%s" aborted build', self.name)
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
298 build.slave = None
69
b92d7c7d70fd Record build slave properties in database.
cmlenz
parents: 66
diff changeset
299 build.started = 0
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
300 build.status = Build.PENDING
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
301
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
302 elif elem.name == 'error':
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
303 build.status = Build.FAILURE
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
304
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
305 build.update()
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
306
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
307 # TODO: should not block while reading the file; rather stream it using
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
308 # asyncore push_with_producer()
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
309 snapshot_path = self.master.get_snapshot(build, type, encoding)
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
310 snapshot_name = os.path.basename(snapshot_path)
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
311 message = beep.Payload(file(snapshot_path).read(),
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
312 content_disposition=snapshot_name,
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
313 content_type=type, content_encoding=encoding)
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
314 self.channel.send_msg(message, handle_reply=handle_reply)
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
315
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
316
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
317 def _parse_iso_datetime(string):
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
318 """Minimal parser for ISO date-time strings.
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
319
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
320 Return the time as floating point number. Only handles UTC timestamps
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
321 without time zone information."""
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
322 try:
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
323 string = string.split('.', 1)[0] # strip out microseconds
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
324 secs = time.mktime(time.strptime(string, '%Y-%m-%dT%H:%M:%S'))
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
325 tzoffset = time.timezone
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
326 if time.daylight:
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
327 tzoffset = time.altzone
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
328 return secs - tzoffset
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
329 except ValueError, e:
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
330 raise ValueError, 'Invalid ISO date/time %s (%s)' % (string, e)
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
331
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
332
31
0b82d012e42c Add thin script shells around master and slave.
cmlenz
parents: 29
diff changeset
333 def main():
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
334 from bitten import __version__ as VERSION
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
335 from optparse import OptionParser
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
336
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
337 parser = OptionParser(usage='usage: %prog [options] env-path',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
338 version='%%prog %s' % VERSION)
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
339 parser.add_option('-p', '--port', action='store', type='int', dest='port',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
340 help='port number to use')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
341 parser.add_option('-H', '--host', action='store', dest='host',
33
d8d44216258a Exit the slave script when the master disconnects; and other minor fixes.
cmlenz
parents: 31
diff changeset
342 help='the host name or IP address to bind to')
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
343 parser.add_option('--debug', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
344 const=logging.DEBUG, help='enable debugging output')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
345 parser.add_option('-v', '--verbose', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
346 const=logging.INFO, help='print as much as possible')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
347 parser.add_option('-q', '--quiet', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
348 const=logging.ERROR, help='print as little as possible')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
349 parser.set_defaults(port=7633, loglevel=logging.WARNING)
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
350 options, args = parser.parse_args()
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
351
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
352 if len(args) < 1:
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
353 parser.error('incorrect number of arguments')
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
354 env_path = args[0]
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
355
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
356 logging.getLogger().setLevel(options.loglevel)
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
357 port = options.port
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
358 if not (1 <= port <= 65535):
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
359 parser.error('port must be an integer in the range 1-65535')
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
360
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
361 host = options.host
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
362 if not host:
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
363 import socket
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
364 ip = socket.gethostbyname(socket.gethostname())
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
365 try:
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
366 host = socket.gethostbyaddr(ip)[0]
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
367 except socket.error, e:
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
368 logging.warning('Reverse host name lookup failed (%s)', e)
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
369 host = ip
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
370
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
371 master = Master(env_path, host, port)
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
372 try:
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
373 master.run(timeout=5.0)
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
374 except KeyboardInterrupt:
34
6da9468a6879 The build master now gracefully exits by first terminating all active sessions. Fixes #7.
cmlenz
parents: 33
diff changeset
375 master.quit()
31
0b82d012e42c Add thin script shells around master and slave.
cmlenz
parents: 29
diff changeset
376
0b82d012e42c Add thin script shells around master and slave.
cmlenz
parents: 29
diff changeset
377 if __name__ == '__main__':
33
d8d44216258a Exit the slave script when the master disconnects; and other minor fixes.
cmlenz
parents: 31
diff changeset
378 main()
Copyright (C) 2012-2017 Edgewall Software