Mercurial > bitten > bitten-test

annotate bitten/master.py @ 61:47ab019508dd

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Moved {{{BuildError}}} class into package {{{bitten.build}}}.
author cmlenz
date Tue, 28 Jun 2005 10:47:07 +0000
parents 055a6c666fa8
children 2332aedba328
rev   line source
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
1 # -*- coding: iso8859-1 -*-
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
2 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
3 # Copyright (C) 2005 Christopher Lenz <cmlenz@gmx.de>
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
4 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
5 # Bitten is free software; you can redistribute it and/or
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
6 # modify it under the terms of the GNU General Public License as
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
7 # published by the Free Software Foundation; either version 2 of the
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
8 # License, or (at your option) any later version.
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
9 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
10 # Trac is distributed in the hope that it will be useful,
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
11 # but WITHOUT ANY WARRANTY; without even the implied warranty of
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
12 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
13 # General Public License for more details.
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
14 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
15 # You should have received a copy of the GNU General Public License
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
16 # along with this program; if not, write to the Free Software
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
17 # Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
18 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
19 # Author: Christopher Lenz <cmlenz@gmx.de>
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
20
15
06207499c58c * Use logging in the BEEP core as well as in the master and slave scripts. Closes #4.
cmlenz
parents: 14
diff changeset
21 import logging
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
22 import os.path
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
23 import time
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
24
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
25 from trac.env import Environment
45
80bc0fae3ed1 Renamed {{{Configuration}}} to {{{BuildConfig}}}.
cmlenz
parents: 42
diff changeset
26 from bitten.model import Build, BuildConfig
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
27 from bitten.util import archive, beep, xmlio
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
28
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
29
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
30 class Master(beep.Listener):
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
31
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
32 TRIGGER_INTERVAL = 10
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
33
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
34 def __init__(self, env_path, ip, port):
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
35 beep.Listener.__init__(self, ip, port)
24
cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels.
cmlenz
parents: 19
diff changeset
36 self.profiles[OrchestrationProfileHandler.URI] = OrchestrationProfileHandler
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
37 self.env = Environment(env_path)
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
38
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
39 self.slaves = {}
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
40
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
41 # path to generated snapshot archives, key is (config name, revision)
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
42 self.snapshots = {}
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
43
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
44 self.schedule(self.TRIGGER_INTERVAL, self._check_build_triggers)
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
45
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
46 def close(self):
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
47 # Remove all pending builds
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
48 for build in Build.select(self.env, status=Build.PENDING):
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
49 build.delete()
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
50 beep.Listener.close(self)
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
51
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
52 def _check_build_triggers(self, master, when):
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
53 self.schedule(self.TRIGGER_INTERVAL, self._check_build_triggers)
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
54
24
cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels.
cmlenz
parents: 19
diff changeset
55 logging.debug('Checking for build triggers...')
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
56 repos = self.env.get_repository()
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
57 try:
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
58 repos.sync()
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
59
45
80bc0fae3ed1 Renamed {{{Configuration}}} to {{{BuildConfig}}}.
cmlenz
parents: 42
diff changeset
60 for config in BuildConfig.select(self.env):
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
61 node = repos.get_node(config.path)
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
62 for path, rev, chg in node.get_history():
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
63 # Check whether the latest revision of that configuration has
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
64 # already been built
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
65 builds = Build.select(self.env, config.name, rev)
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
66 if not list(builds):
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
67 logging.info('Enqueuing build of configuration "%s" as '
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
68 'of revision [%s]', config.name, rev)
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
69 build = Build(self.env)
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
70 build.config = config.name
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
71
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
72 chgset = repos.get_changeset(rev)
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
73 build.rev = rev
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
74 build.rev_time = chgset.date
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
75
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
76 build.insert()
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
77 break
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
78 finally:
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
79 repos.close()
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
80
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
81 self.schedule(5, self._check_build_queue)
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
82
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
83 def _check_build_queue(self, master, when):
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
84 if not self.slaves:
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
85 return
49
b4c51e32952b Build slave now also handles snapshot archives in bzip2 and zip format.
cmlenz
parents: 48
diff changeset
86 logging.debug('Checking for pending builds...')
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
87 for build in Build.select(self.env, status=Build.PENDING):
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
88 for slave in self.slaves.values():
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
89 active_builds = Build.select(self.env, slave=slave.name,
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
90 status=Build.IN_PROGRESS)
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
91 if not list(active_builds):
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
92 slave.send_initiation(build)
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
93 break
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
94
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
95 def get_snapshot(self, build, type, encoding):
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
96 formats = {
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
97 ('application/tar', 'bzip2'): 'bzip2',
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
98 ('application/tar', 'gzip'): 'bzip',
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
99 ('application/tar', None): 'tar',
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
100 ('application/zip', None): 'zip',
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
101 }
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
102 if not (build.config, build.rev, type, encoding) in self.snapshots:
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
103 config = BuildConfig(self.env, build.config)
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
104 snapshot = archive.pack(self.env, path=config.path, rev=build.rev,
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
105 prefix=config.name,
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
106 format=formats[(type, encoding)])
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
107 logging.info('Prepared snapshot archive at %s' % snapshot)
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
108 self.snapshots[(build.config, build.rev, type, encoding)] = snapshot
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
109 return self.snapshots[(build.config, build.rev, type, encoding)]
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
110
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
111
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
112 class OrchestrationProfileHandler(beep.ProfileHandler):
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
113 """Handler for communication on the Bitten build orchestration profile from
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
114 the perspective of the build master.
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
115 """
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
116 URI = 'http://bitten.cmlenz.net/beep/orchestration'
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
117
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
118 def handle_connect(self):
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
119 self.master = self.session.listener
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
120 assert self.master
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
121 self.name = None
24
cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels.
cmlenz
parents: 19
diff changeset
122
cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels.
cmlenz
parents: 19
diff changeset
123 def handle_disconnect(self):
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
124 del self.master.slaves[self.name]
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
125
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
126 for build in Build.select(self.master.env, slave=self.name,
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
127 status=Build.IN_PROGRESS):
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
128 logging.info('Build [%s] of "%s" by %s cancelled', build.rev,
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
129 build.config, self.name)
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
130 build.slave = None
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
131 build.status = Build.PENDING
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
132 build.time = None
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
133 build.update()
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
134 break
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
135 logging.info('Unregistered slave "%s"', self.name)
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
136
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
137 def handle_msg(self, msgno, msg):
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
138 assert msg.get_content_type() == beep.BEEP_XML
29
2adc3480e4aa Some cleanup and additional docstrings.
cmlenz
parents: 24
diff changeset
139 elem = xmlio.parse(msg.get_payload())
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
140
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
141 if elem.name == 'register':
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
142 platform, os, os_family, os_version = None, None, None, None
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
143 for child in elem.children():
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
144 if child.name == 'platform':
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
145 platform = child.gettext()
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
146 elif child.name == 'os':
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
147 os = child.gettext()
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
148 os_family = child.attr['family']
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
149 os_version = child.attr['version']
14
1733c601d2f8 Refactored the asyncore loop and shutdown procedure into {{{beep.Initiator}}}.
cmlenz
parents: 13
diff changeset
150
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
151 self.name = elem.attr['name']
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
152 self.master.slaves[self.name] = self
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
153
29
2adc3480e4aa Some cleanup and additional docstrings.
cmlenz
parents: 24
diff changeset
154 xml = xmlio.Element('ok')
2adc3480e4aa Some cleanup and additional docstrings.
cmlenz
parents: 24
diff changeset
155 self.channel.send_rpy(msgno, beep.MIMEMessage(xml))
24
cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels.
cmlenz
parents: 19
diff changeset
156 logging.info('Registered slave "%s" (%s running %s %s [%s])',
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
157 self.name, platform, os, os_version, os_family)
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
158
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
159 def send_initiation(self, build):
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
160 logging.debug('Initiating build of "%s" on slave %s', build.config,
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
161 self.name)
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
162
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
163 def handle_reply(cmd, msgno, ansno, msg):
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
164 if cmd == 'ERR':
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
165 if msg.get_content_type() == beep.BEEP_XML:
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
166 elem = xmlio.parse(msg.get_payload())
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
167 if elem.name == 'error':
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
168 logging.warning('Slave refused build request: %s (%d)',
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
169 elem.gettext(), int(elem.attr['code']))
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
170 return
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
171
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
172 elem = xmlio.parse(msg.get_payload())
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
173 assert elem.name == 'proceed'
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
174 type = encoding = None
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
175 for child in elem.children('accept'):
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
176 type, encoding = child.attr['type'], child.attr.get('encoding')
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
177 if (type, encoding) in (('application/tar', 'gzip'),
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
178 ('application/tar', 'bzip2'),
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
179 ('application/tar', None),
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
180 ('application/zip', None)):
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
181 break
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
182 type = None
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
183 if not type:
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
184 xml = xmlio.Element('error', code=550)[
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
185 'None of the supported archive formats accepted'
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
186 ]
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
187 self.channel.send_err(beep.MIMEMessage(xml))
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
188 return
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
189 self.send_snapshot(build, type, encoding)
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
190
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
191 xml = xmlio.Element('build', recipe='recipe.xml')
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
192 self.channel.send_msg(beep.MIMEMessage(xml), handle_reply=handle_reply)
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
193
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
194 def send_snapshot(self, build, type, encoding):
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
195 def handle_reply(cmd, msgno, ansno, msg):
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
196 if cmd == 'ERR':
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
197 if msg.get_content_type() == beep.BEEP_XML:
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
198 elem = xmlio.parse(msg.get_payload())
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
199 if elem.name == 'error':
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
200 logging.warning('Slave did not accept archive: %s (%d)',
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
201 elem.gettext(), int(elem.attr['code']))
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
202 return
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
203 if cmd == 'ANS':
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
204 if ansno == 0:
60
055a6c666fa8 * Pass a {{{Context}}} object to recipe commands as the first argument. Currently this only has the basedir, but will be extended to also provide output recording etc.
cmlenz
parents: 56
diff changeset
205 self.steps = []
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
206 build.slave = self.name
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
207 build.time = int(time.time())
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
208 build.status = Build.IN_PROGRESS
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
209 build.update()
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
210 logging.info('Slave %s started build of "%s" as of [%s]',
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
211 self.name, build.config, build.rev)
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
212 else:
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
213 elem = xmlio.parse(msg.get_payload())
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
214 assert elem.name == 'step'
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
215 logging.info('Slave completed step "%s"', elem.attr['id'])
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
216 if elem.attr['result'] == 'failure':
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
217 logging.warning('Step failed: %s', elem.gettext())
60
055a6c666fa8 * Pass a {{{Context}}} object to recipe commands as the first argument. Currently this only has the basedir, but will be extended to also provide output recording etc.
cmlenz
parents: 56
diff changeset
218 self.steps.append((elem.attr['id'], elem.attr['result']))
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
219 elif cmd == 'NUL':
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
220 logging.info('Slave %s completed build of "%s" as of [%s]',
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
221 self.name, build.config, build.rev)
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
222 build.duration = int(time.time()) - build.time
60
055a6c666fa8 * Pass a {{{Context}}} object to recipe commands as the first argument. Currently this only has the basedir, but will be extended to also provide output recording etc.
cmlenz
parents: 56
diff changeset
223 if [step for step in self.steps if step[1] == 'failure']:
055a6c666fa8 * Pass a {{{Context}}} object to recipe commands as the first argument. Currently this only has the basedir, but will be extended to also provide output recording etc.
cmlenz
parents: 56
diff changeset
224 build.status = Build.FAILURE
055a6c666fa8 * Pass a {{{Context}}} object to recipe commands as the first argument. Currently this only has the basedir, but will be extended to also provide output recording etc.
cmlenz
parents: 56
diff changeset
225 else:
055a6c666fa8 * Pass a {{{Context}}} object to recipe commands as the first argument. Currently this only has the basedir, but will be extended to also provide output recording etc.
cmlenz
parents: 56
diff changeset
226 build.status = Build.SUCCESS
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
227 build.update()
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
228
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
229 # TODO: should not block while reading the file; rather stream it using
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
230 # asyncore push_with_producer()
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
231 snapshot_path = self.master.get_snapshot(build, type, encoding)
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
232 snapshot_name = os.path.basename(snapshot_path)
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
233 message = beep.MIMEMessage(file(snapshot_path).read(),
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
234 content_disposition=snapshot_name,
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
235 content_type=type, content_encoding=encoding)
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
236 self.channel.send_msg(message, handle_reply=handle_reply)
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
237
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
238
31
0b82d012e42c Add thin script shells around master and slave.
cmlenz
parents: 29
diff changeset
239 def main():
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
240 from bitten import __version__ as VERSION
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
241 from optparse import OptionParser
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
242
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
243 parser = OptionParser(usage='usage: %prog [options] env-path',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
244 version='%%prog %s' % VERSION)
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
245 parser.add_option('-p', '--port', action='store', type='int', dest='port',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
246 help='port number to use')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
247 parser.add_option('-H', '--host', action='store', dest='host',
33
d8d44216258a Exit the slave script when the master disconnects; and other minor fixes.
cmlenz
parents: 31
diff changeset
248 help='the host name or IP address to bind to')
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
249 parser.add_option('--debug', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
250 const=logging.DEBUG, help='enable debugging output')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
251 parser.add_option('-v', '--verbose', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
252 const=logging.INFO, help='print as much as possible')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
253 parser.add_option('-q', '--quiet', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
254 const=logging.ERROR, help='print as little as possible')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
255 parser.set_defaults(port=7633, loglevel=logging.WARNING)
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
256 options, args = parser.parse_args()
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
257
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
258 if len(args) < 1:
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
259 parser.error('incorrect number of arguments')
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
260 env_path = args[0]
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
261
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
262 logging.getLogger().setLevel(options.loglevel)
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
263 port = options.port
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
264 if not (1 <= port <= 65535):
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
265 parser.error('port must be an integer in the range 1-65535')
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
266
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
267 host = options.host
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
268 if not host:
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
269 import socket
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
270 ip = socket.gethostbyname(socket.gethostname())
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
271 try:
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
272 host = socket.gethostbyaddr(ip)[0]
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
273 except socket.error, e:
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
274 logging.warning('Reverse host name lookup failed (%s)', e)
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
275 host = ip
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
276
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
277 master = Master(env_path, host, port)
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
278 try:
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
279 master.run(timeout=5.0)
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
280 except KeyboardInterrupt:
34
6da9468a6879 The build master now gracefully exits by first terminating all active sessions. Fixes #7.
cmlenz
parents: 33
diff changeset
281 master.quit()
31
0b82d012e42c Add thin script shells around master and slave.
cmlenz
parents: 29
diff changeset
282
0b82d012e42c Add thin script shells around master and slave.
cmlenz
parents: 29
diff changeset
283 if __name__ == '__main__':
33
d8d44216258a Exit the slave script when the master disconnects; and other minor fixes.
cmlenz
parents: 31
diff changeset
284 main()
Copyright (C) 2012-2017 Edgewall Software