Mercurial > bitten > bitten-test

annotate bitten/master.py @ 96:c8c36f34ff5a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Change model class pattern: * Using normal constructor creates a new object, that can added to the database using the {{{insert()}}} method. * For getting an existing object, the class method {{{fetch()}}} must be used. It's similar to {{{select()}}}, but returns only one object.
author cmlenz
date Sun, 17 Jul 2005 20:14:43 +0000
parents 1984b2e01998
children 5bf22bb87915
rev   line source
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
1 # -*- coding: iso8859-1 -*-
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
2 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
3 # Copyright (C) 2005 Christopher Lenz <cmlenz@gmx.de>
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
4 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
5 # Bitten is free software; you can redistribute it and/or
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
6 # modify it under the terms of the GNU General Public License as
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
7 # published by the Free Software Foundation; either version 2 of the
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
8 # License, or (at your option) any later version.
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
9 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
10 # Trac is distributed in the hope that it will be useful,
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
11 # but WITHOUT ANY WARRANTY; without even the implied warranty of
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
12 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
13 # General Public License for more details.
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
14 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
15 # You should have received a copy of the GNU General Public License
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
16 # along with this program; if not, write to the Free Software
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
17 # Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
18 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
19 # Author: Christopher Lenz <cmlenz@gmx.de>
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
20
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
21 from datetime import datetime, timedelta
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
22 from itertools import ifilter
15
06207499c58c * Use logging in the BEEP core as well as in the master and slave scripts. Closes #4.
cmlenz
parents: 14
diff changeset
23 import logging
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
24 import os.path
76
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
25 import re
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
26 try:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
27 set
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
28 except NameError:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
29 from sets import Set as set
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
30 import time
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
31
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
32 from trac.env import Environment
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
33 from bitten.model import BuildConfig, TargetPlatform, Build, BuildStep
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
34 from bitten.util import archive, beep, xmlio
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
35
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
36 log = logging.getLogger('bitten.master')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
37
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
38 DEFAULT_CHECK_INTERVAL = 120 # 2 minutes
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
39
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
40
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
41
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
42 class Master(beep.Listener):
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
43
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
44 def __init__(self, env_path, ip, port,
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
45 check_interval=DEFAULT_CHECK_INTERVAL):
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
46 beep.Listener.__init__(self, ip, port)
24
cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels.
cmlenz
parents: 19
diff changeset
47 self.profiles[OrchestrationProfileHandler.URI] = OrchestrationProfileHandler
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
48 self.env = Environment(env_path)
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
49 self.check_interval = check_interval
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
50
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
51 self.slaves = {}
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
52
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
53 # path to generated snapshot archives, key is (config name, revision)
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
54 self.snapshots = {}
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
55 for config in BuildConfig.select(self.env):
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
56 snapshots = archive.index(self.env, prefix=config.name)
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
57 for rev, format, path in snapshots:
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
58 self.snapshots[(config.name, rev, format)] = path
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
59
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
60 self.schedule(self.check_interval, self._check_build_triggers)
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
61
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
62 def close(self):
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
63 # Remove all pending builds
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
64 for build in Build.select(self.env, status=Build.PENDING):
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
65 build.delete()
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
66 beep.Listener.close(self)
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
67
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
68 def _check_build_triggers(self, when):
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
69 self.schedule(self.check_interval, self._check_build_triggers)
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
70
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
71 repos = self.env.get_repository()
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
72 try:
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
73 repos.sync()
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
74
45
80bc0fae3ed1 Renamed {{{Configuration}}} to {{{BuildConfig}}}.
cmlenz
parents: 42
diff changeset
75 for config in BuildConfig.select(self.env):
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
76 log.debug('Checking for changes to "%s" at %s', config.label,
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
77 config.path)
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
78 node = repos.get_node(config.path)
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
79 for path, rev, chg in node.get_history():
76
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
80 enqueued = False
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
81 for platform in TargetPlatform.select(self.env, config.name):
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
82 # Check whether the latest revision of the configuration
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
83 # has already been built on this platform
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
84 builds = Build.select(self.env, config.name, rev,
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
85 platform.id)
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
86 if not list(builds):
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
87 log.info('Enqueuing build of configuration "%s" at '
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
88 'revision [%s] on %s', config.name, rev,
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
89 platform.name)
76
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
90 build = Build(self.env)
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
91 build.config = config.name
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
92 build.rev = str(rev)
76
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
93 build.rev_time = repos.get_changeset(rev).date
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
94 build.platform = platform.id
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
95 build.insert()
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
96 enqueued = True
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
97 if enqueued:
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
98 break
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
99 finally:
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
100 repos.close()
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
101
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
102 self.schedule(self.check_interval * 0.2, self._check_build_queue)
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
103 self.schedule(self.check_interval * 1.8, self._cleanup_snapshots)
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
104
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
105 def _check_build_queue(self, when):
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
106 if not self.slaves:
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
107 return
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
108 log.debug('Checking for pending builds...')
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
109 for build in Build.select(self.env, status=Build.PENDING):
86
110bfa3cbc32 Slaves were getting associated with the wrong target platform.
cmlenz
parents: 85
diff changeset
110 for slave in self.slaves.get(build.platform, []):
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
111 active_builds = Build.select(self.env, slave=slave.name,
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
112 status=Build.IN_PROGRESS)
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
113 if not list(active_builds):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
114 slave.send_initiation(build)
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
115 return
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
116
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
117 def _cleanup_snapshots(self, when):
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
118 log.debug('Checking for unused snapshot archives...')
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
119 for (config, rev, format), path in self.snapshots.items():
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
120 keep = False
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
121 for build in Build.select(self.env, config=config, rev=rev):
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
122 if build.status not in (Build.SUCCESS, Build.FAILURE):
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
123 keep = True
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
124 break
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
125 if not keep:
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
126 log.info('Removing unused snapshot %s', path)
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
127 os.unlink(path)
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
128 del self.snapshots[(config, rev, format)]
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
129
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
130 def get_snapshot(self, build, format):
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
131 snapshot = self.snapshots.get((build.config, build.rev, format))
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
132 if not snapshot:
96
c8c36f34ff5a Change model class pattern:
cmlenz
parents: 95
diff changeset
133 config = BuildConfig.fetch(self.env, build.config)
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
134 snapshot = archive.pack(self.env, path=config.path, rev=build.rev,
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
135 prefix=config.name, format=format)
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
136 log.info('Prepared snapshot archive at %s' % snapshot)
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
137 self.snapshots[(build.config, build.rev, format)] = snapshot
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
138 return snapshot
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
139
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
140 def register(self, handler):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
141 any_match = False
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
142 for config in BuildConfig.select(self.env):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
143 for platform in TargetPlatform.select(self.env, config=config.name):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
144 if not platform.id in self.slaves:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
145 self.slaves[platform.id] = set()
85
acb7b67b8152 Fix matching of slave properties against target platform rules.
cmlenz
parents: 84
diff changeset
146 match = True
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
147 for property, pattern in ifilter(None, platform.rules):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
148 try:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
149 if not re.match(pattern, handler.info.get(property)):
86
110bfa3cbc32 Slaves were getting associated with the wrong target platform.
cmlenz
parents: 85
diff changeset
150 match = False
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
151 break
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
152 except re.error, e:
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
153 log.error('Invalid platform matching pattern "%s"',
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
154 pattern, exc_info=True)
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
155 match = False
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
156 break
84
30d09249de80 Fix var name in slave registration.
cmlenz
parents: 83
diff changeset
157 if match:
96
c8c36f34ff5a Change model class pattern:
cmlenz
parents: 95
diff changeset
158 log.debug('Slave %s matched target platform %s',
c8c36f34ff5a Change model class pattern:
cmlenz
parents: 95
diff changeset
159 handler.name, platform.name)
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
160 self.slaves[platform.id].add(handler)
86
110bfa3cbc32 Slaves were getting associated with the wrong target platform.
cmlenz
parents: 85
diff changeset
161 any_match = True
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
162
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
163 if not any_match:
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
164 log.warning('Slave %s does not match any of the configured target '
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
165 'platforms', handler.name)
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
166 return False
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
167
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
168 self.schedule(self.check_interval * 0.2, self._check_build_queue)
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
169
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
170 log.info('Registered slave "%s"', handler.name)
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
171 return True
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
172
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
173 def unregister(self, handler):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
174 for slaves in self.slaves.values():
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
175 slaves.discard(handler)
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
176
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
177 for build in Build.select(self.env, slave=handler.name,
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
178 status=Build.IN_PROGRESS):
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
179 log.info('Build [%s] of "%s" by %s cancelled', build.rev,
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
180 build.config, handler.name)
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
181 build.slave = None
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
182 build.status = Build.PENDING
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
183 build.started = 0
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
184 build.update()
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
185 break
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
186 log.info('Unregistered slave "%s"', handler.name)
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
187
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
188
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
189 class OrchestrationProfileHandler(beep.ProfileHandler):
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
190 """Handler for communication on the Bitten build orchestration profile from
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
191 the perspective of the build master.
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
192 """
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
193 URI = 'http://bitten.cmlenz.net/beep/orchestration'
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
194
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
195 def handle_connect(self):
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
196 self.master = self.session.listener
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
197 assert self.master
69
b92d7c7d70fd Record build slave properties in database.
cmlenz
parents: 66
diff changeset
198 self.env = self.master.env
b92d7c7d70fd Record build slave properties in database.
cmlenz
parents: 66
diff changeset
199 assert self.env
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
200 self.name = None
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
201 self.info = {}
24
cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels.
cmlenz
parents: 19
diff changeset
202
cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels.
cmlenz
parents: 19
diff changeset
203 def handle_disconnect(self):
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
204 self.master.unregister(self)
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
205
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
206 def handle_msg(self, msgno, payload):
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
207 assert payload.content_type == beep.BEEP_XML
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
208 elem = xmlio.parse(payload.body)
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
209
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
210 if elem.name == 'register':
69
b92d7c7d70fd Record build slave properties in database.
cmlenz
parents: 66
diff changeset
211 self.name = elem.attr['name']
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
212 for child in elem.children():
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
213 if child.name == 'platform':
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
214 self.info[Build.MACHINE] = child.gettext()
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
215 self.info[Build.PROCESSOR] = child.attr.get('processor')
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
216 elif child.name == 'os':
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
217 self.info[Build.OS_NAME] = child.gettext()
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
218 self.info[Build.OS_FAMILY] = child.attr.get('family')
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
219 self.info[Build.OS_VERSION] = child.attr.get('version')
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
220 self.info[Build.IP_ADDRESS] = self.session.addr[0]
14
1733c601d2f8 Refactored the asyncore loop and shutdown procedure into {{{beep.Initiator}}}.
cmlenz
parents: 13
diff changeset
221
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
222 if not self.master.register(self):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
223 xml = xmlio.Element('error', code=550)[
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
224 'Nothing for you to build here, please move along'
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
225 ]
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
226 self.channel.send_err(msgno, beep.Payload(xml))
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
227 return
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
228
29
2adc3480e4aa Some cleanup and additional docstrings.
cmlenz
parents: 24
diff changeset
229 xml = xmlio.Element('ok')
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
230 self.channel.send_rpy(msgno, beep.Payload(xml))
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
231
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
232 def send_initiation(self, build):
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
233 log.debug('Initiating build of "%s" on slave %s', build.config,
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
234 self.name)
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
235
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
236 def handle_reply(cmd, msgno, ansno, payload):
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
237 if cmd == 'ERR':
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
238 if payload.content_type == beep.BEEP_XML:
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
239 elem = xmlio.parse(payload.body)
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
240 if elem.name == 'error':
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
241 log.warning('Slave %s refused build request: %s (%d)',
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
242 self.name, elem.gettext(),
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
243 int(elem.attr['code']))
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
244 return
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
245
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
246 elem = xmlio.parse(payload.body)
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
247 assert elem.name == 'proceed'
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
248 type = encoding = None
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
249 for child in elem.children('accept'):
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
250 type, encoding = child.attr['type'], child.attr.get('encoding')
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
251 if (type, encoding) in (('application/tar', 'gzip'),
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
252 ('application/tar', 'bzip2'),
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
253 ('application/tar', None),
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
254 ('application/zip', None)):
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
255 break
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
256 type = None
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
257 if not type:
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
258 xml = xmlio.Element('error', code=550)[
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
259 'None of the accepted archive formats supported'
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
260 ]
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
261 self.channel.send_err(beep.Payload(xml))
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
262 return
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
263 self.send_snapshot(build, type, encoding)
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
264
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
265 xml = xmlio.Element('build', recipe='recipe.xml')
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
266 self.channel.send_msg(beep.Payload(xml), handle_reply=handle_reply)
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
267
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
268 def send_snapshot(self, build, type, encoding):
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
269
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
270 def handle_reply(cmd, msgno, ansno, payload):
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
271 if cmd == 'ERR':
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
272 assert payload.content_type == beep.BEEP_XML
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
273 elem = xmlio.parse(payload.body)
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
274 if elem.name == 'error':
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
275 log.warning('Slave %s did not accept archive: %s (%d)',
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
276 self.name, elem.gettext(),
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
277 int(elem.attr['code']))
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
278
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
279 if cmd == 'ANS':
94
e5d1a792aa45 Cleanup and records when a build is aborted.
cmlenz
parents: 93
diff changeset
280 db = self.env.get_db_cnx()
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
281 elem = xmlio.parse(payload.body)
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
282
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
283 if elem.name == 'started':
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
284 build.slave = self.name
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
285 build.slave_info.update(self.info)
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
286 build.started = int(_parse_iso_datetime(elem.attr['time']))
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
287 build.status = Build.IN_PROGRESS
96
c8c36f34ff5a Change model class pattern:
cmlenz
parents: 95
diff changeset
288 log.info('Slave %s started build %d ("%s" as of [%s])',
c8c36f34ff5a Change model class pattern:
cmlenz
parents: 95
diff changeset
289 self.name, build.id, build.config, build.rev)
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
290
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
291 elif elem.name == 'step':
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
292 log.info('Slave completed step "%s"', elem.attr['id'])
96
c8c36f34ff5a Change model class pattern:
cmlenz
parents: 95
diff changeset
293 step = BuildStep(self.env, build=build.id,
c8c36f34ff5a Change model class pattern:
cmlenz
parents: 95
diff changeset
294 name=elem.attr['id'],
c8c36f34ff5a Change model class pattern:
cmlenz
parents: 95
diff changeset
295 description=elem.attr.get('description'),
c8c36f34ff5a Change model class pattern:
cmlenz
parents: 95
diff changeset
296 log=elem.gettext().strip())
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
297 step.started = int(_parse_iso_datetime(elem.attr['time']))
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
298 step.stopped = step.started + int(elem.attr['duration'])
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
299 if elem.attr['result'] == 'failure':
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
300 log.warning('Step failed: %s', elem.gettext())
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
301 step.status = BuildStep.FAILURE
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
302 else:
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
303 step.status = BuildStep.SUCCESS
94
e5d1a792aa45 Cleanup and records when a build is aborted.
cmlenz
parents: 93
diff changeset
304 step.insert(db=db)
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
305
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
306 elif elem.name == 'completed':
96
c8c36f34ff5a Change model class pattern:
cmlenz
parents: 95
diff changeset
307 log.info('Slave %s completed build %d ("%s" as of [%s])',
c8c36f34ff5a Change model class pattern:
cmlenz
parents: 95
diff changeset
308 self.name, build.id, build.config, build.rev)
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
309 build.stopped = int(_parse_iso_datetime(elem.attr['time']))
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
310 if elem.attr['result'] == 'failure':
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
311 build.status = Build.FAILURE
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
312 else:
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
313 build.status = Build.SUCCESS
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
314
70
ccd03b6f04ef Updated DTD for orchestration profile.
cmlenz
parents: 69
diff changeset
315 elif elem.name == 'aborted':
96
c8c36f34ff5a Change model class pattern:
cmlenz
parents: 95
diff changeset
316 log.info('Slave "%s" aborted build %d ("%s" as of [%s])',
c8c36f34ff5a Change model class pattern:
cmlenz
parents: 95
diff changeset
317 self.name, build.id, build.config, build.rev)
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
318 build.slave = None
69
b92d7c7d70fd Record build slave properties in database.
cmlenz
parents: 66
diff changeset
319 build.started = 0
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
320 build.status = Build.PENDING
94
e5d1a792aa45 Cleanup and records when a build is aborted.
cmlenz
parents: 93
diff changeset
321 build.slave_info = {}
e5d1a792aa45 Cleanup and records when a build is aborted.
cmlenz
parents: 93
diff changeset
322 for step in BuildStep.select(self.env, build=build.id,
e5d1a792aa45 Cleanup and records when a build is aborted.
cmlenz
parents: 93
diff changeset
323 db=db):
e5d1a792aa45 Cleanup and records when a build is aborted.
cmlenz
parents: 93
diff changeset
324 step.delete(db=db)
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
325
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
326 elif elem.name == 'error':
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
327 build.status = Build.FAILURE
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
328
94
e5d1a792aa45 Cleanup and records when a build is aborted.
cmlenz
parents: 93
diff changeset
329 build.update(db=db)
e5d1a792aa45 Cleanup and records when a build is aborted.
cmlenz
parents: 93
diff changeset
330 db.commit()
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
331
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
332 snapshot_format = {
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
333 ('application/tar', 'bzip2'): 'bzip2',
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
334 ('application/tar', 'gzip'): 'gzip',
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
335 ('application/tar', None): 'tar',
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
336 ('application/zip', None): 'zip',
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
337 }[(type, encoding)]
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
338 snapshot_path = self.master.get_snapshot(build, snapshot_format)
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
339 snapshot_name = os.path.basename(snapshot_path)
90
2c4e104afef8 The build master now transmits snapshot archives without blocking while reading the file and sending the BEEP frames. Closes #17.
cmlenz
parents: 88
diff changeset
340 message = beep.Payload(file(snapshot_path), content_type=type,
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
341 content_disposition=snapshot_name,
90
2c4e104afef8 The build master now transmits snapshot archives without blocking while reading the file and sending the BEEP frames. Closes #17.
cmlenz
parents: 88
diff changeset
342 content_encoding=encoding)
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
343 self.channel.send_msg(message, handle_reply=handle_reply)
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
344
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
345
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
346 def _parse_iso_datetime(string):
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
347 """Minimal parser for ISO date-time strings.
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
348
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
349 Return the time as floating point number. Only handles UTC timestamps
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
350 without time zone information."""
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
351 try:
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
352 string = string.split('.', 1)[0] # strip out microseconds
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
353 secs = time.mktime(time.strptime(string, '%Y-%m-%dT%H:%M:%S'))
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
354 tzoffset = time.timezone
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
355 if time.daylight:
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
356 tzoffset = time.altzone
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
357 return secs - tzoffset
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
358 except ValueError, e:
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
359 raise ValueError, 'Invalid ISO date/time %s (%s)' % (string, e)
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
360
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
361
31
0b82d012e42c Add thin script shells around master and slave.
cmlenz
parents: 29
diff changeset
362 def main():
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
363 from bitten import __version__ as VERSION
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
364 from optparse import OptionParser
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
365
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
366 parser = OptionParser(usage='usage: %prog [options] env-path',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
367 version='%%prog %s' % VERSION)
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
368 parser.add_option('-p', '--port', action='store', type='int', dest='port',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
369 help='port number to use')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
370 parser.add_option('-H', '--host', action='store', dest='host',
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
371 metavar='HOSTNAME',
33
d8d44216258a Exit the slave script when the master disconnects; and other minor fixes.
cmlenz
parents: 31
diff changeset
372 help='the host name or IP address to bind to')
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
373 parser.add_option('-l', '--log', dest='logfile', metavar='FILENAME',
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
374 help='write log messages to FILENAME')
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
375 parser.add_option('-i', '--interval', dest='interval', metavar='SECONDS',
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
376 default=DEFAULT_CHECK_INTERVAL, type='int',
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
377 help='poll interval for changeset detection')
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
378 parser.add_option('--debug', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
379 const=logging.DEBUG, help='enable debugging output')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
380 parser.add_option('-v', '--verbose', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
381 const=logging.INFO, help='print as much as possible')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
382 parser.add_option('-q', '--quiet', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
383 const=logging.ERROR, help='print as little as possible')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
384 parser.set_defaults(port=7633, loglevel=logging.WARNING)
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
385 options, args = parser.parse_args()
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
386
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
387 if len(args) < 1:
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
388 parser.error('incorrect number of arguments')
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
389 env_path = args[0]
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
390
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
391 # Configure logging
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
392 log = logging.getLogger('bitten')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
393 log.setLevel(options.loglevel)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
394 handler = logging.StreamHandler()
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
395 if options.logfile:
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
396 handler.setLevel(logging.WARNING)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
397 else:
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
398 handler.setLevel(options.loglevel)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
399 formatter = logging.Formatter('%(message)s')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
400 handler.setFormatter(formatter)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
401 log.addHandler(handler)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
402 if options.logfile:
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
403 handler = logging.FileHandler(options.logfile)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
404 handler.setLevel(options.loglevel)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
405 formatter = logging.Formatter('%(asctime)s [%(name)s] %(levelname)s: '
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
406 '%(message)s')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
407 handler.setFormatter(formatter)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
408 log.addHandler(handler)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
409
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
410 port = options.port
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
411 if not (1 <= port <= 65535):
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
412 parser.error('port must be an integer in the range 1-65535')
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
413
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
414 host = options.host
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
415 if not host:
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
416 import socket
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
417 ip = socket.gethostbyname(socket.gethostname())
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
418 try:
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
419 host = socket.gethostbyaddr(ip)[0]
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
420 except socket.error, e:
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
421 log.warning('Reverse host name lookup failed (%s)', e)
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
422 host = ip
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
423
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
424 master = Master(env_path, host, port, check_interval=options.interval)
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
425 try:
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
426 master.run(timeout=5.0)
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
427 except KeyboardInterrupt:
34
6da9468a6879 The build master now gracefully exits by first terminating all active sessions. Fixes #7.
cmlenz
parents: 33
diff changeset
428 master.quit()
31
0b82d012e42c Add thin script shells around master and slave.
cmlenz
parents: 29
diff changeset
429
0b82d012e42c Add thin script shells around master and slave.
cmlenz
parents: 29
diff changeset
430 if __name__ == '__main__':
33
d8d44216258a Exit the slave script when the master disconnects; and other minor fixes.
cmlenz
parents: 31
diff changeset
431 main()
Copyright (C) 2012-2017 Edgewall Software