Mercurial > bitten > bitten-test

annotate bitten/master.py @ 112:a38eabd4b6e1

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
* Store build logs in a structured way, for example to highlight messages on the error stream. * Add basic infrastructure for database upgrades.
author cmlenz
date Thu, 04 Aug 2005 20:15:39 +0000
parents c58c74e7df16
children 16d69eb6e047
rev   line source
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
1 # -*- coding: iso8859-1 -*-
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
2 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
3 # Copyright (C) 2005 Christopher Lenz <cmlenz@gmx.de>
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
4 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
5 # Bitten is free software; you can redistribute it and/or
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
6 # modify it under the terms of the GNU General Public License as
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
7 # published by the Free Software Foundation; either version 2 of the
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
8 # License, or (at your option) any later version.
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
9 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
10 # Trac is distributed in the hope that it will be useful,
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
11 # but WITHOUT ANY WARRANTY; without even the implied warranty of
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
12 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
13 # General Public License for more details.
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
14 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
15 # You should have received a copy of the GNU General Public License
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
16 # along with this program; if not, write to the Free Software
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
17 # Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
18 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
19 # Author: Christopher Lenz <cmlenz@gmx.de>
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
20
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
21 from datetime import datetime, timedelta
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
22 from itertools import ifilter
15
06207499c58c * Use logging in the BEEP core as well as in the master and slave scripts. Closes #4.
cmlenz
parents: 14
diff changeset
23 import logging
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
24 import os.path
76
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
25 import re
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
26 try:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
27 set
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
28 except NameError:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
29 from sets import Set as set
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
30 import time
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
31
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
32 from trac.env import Environment
112
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
33 from bitten.model import BuildConfig, TargetPlatform, Build, BuildStep, BuildLog
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
34 from bitten.util import archive, beep, xmlio
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
35
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
36 log = logging.getLogger('bitten.master')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
37
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
38 DEFAULT_CHECK_INTERVAL = 120 # 2 minutes
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
39
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
40
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
41 class Master(beep.Listener):
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
42
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
43 def __init__(self, env_path, ip, port,
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
44 check_interval=DEFAULT_CHECK_INTERVAL):
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
45 beep.Listener.__init__(self, ip, port)
24
cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels.
cmlenz
parents: 19
diff changeset
46 self.profiles[OrchestrationProfileHandler.URI] = OrchestrationProfileHandler
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
47 self.env = Environment(env_path)
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
48 self.check_interval = check_interval
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
49
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
50 self.slaves = {}
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
51
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
52 # path to generated snapshot archives, key is (config name, revision)
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
53 self.snapshots = {}
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
54 for config in BuildConfig.select(self.env):
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
55 snapshots = archive.index(self.env, prefix=config.name)
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
56 for rev, format, path in snapshots:
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
57 self.snapshots[(config.name, rev, format)] = path
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
58
109
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
59 self._cleanup_orphaned_builds()
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
60 self.schedule(self.check_interval, self._check_build_triggers)
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
61
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
62 def close(self):
109
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
63 self._cleanup_orphaned_builds()
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
64 beep.Listener.close(self)
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
65
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
66 def _check_build_triggers(self, when):
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
67 self.schedule(self.check_interval, self._check_build_triggers)
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
68
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
69 repos = self.env.get_repository()
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
70 try:
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
71 repos.sync()
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
72
112
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
73 db = self.env.get_db_cnx()
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
74 for config in BuildConfig.select(self.env, db=db):
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
75 log.debug('Checking for changes to "%s" at %s', config.label,
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
76 config.path)
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
77 node = repos.get_node(config.path)
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
78 for path, rev, chg in node.get_history():
76
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
79 enqueued = False
112
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
80 for platform in TargetPlatform.select(self.env,
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
81 config.name, db=db):
76
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
82 # Check whether the latest revision of the configuration
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
83 # has already been built on this platform
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
84 builds = Build.select(self.env, config.name, rev,
112
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
85 platform.id, db=db)
76
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
86 if not list(builds):
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
87 log.info('Enqueuing build of configuration "%s" at '
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
88 'revision [%s] on %s', config.name, rev,
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
89 platform.name)
76
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
90 build = Build(self.env)
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
91 build.config = config.name
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
92 build.rev = str(rev)
76
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
93 build.rev_time = repos.get_changeset(rev).date
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
94 build.platform = platform.id
112
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
95 build.insert(db)
76
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
96 enqueued = True
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15.
cmlenz
parents: 73
diff changeset
97 if enqueued:
112
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
98 db.commit()
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
99 break
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
100 finally:
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
101 repos.close()
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
102
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
103 self.schedule(self.check_interval * 0.2, self._check_build_queue)
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
104 self.schedule(self.check_interval * 1.8, self._cleanup_snapshots)
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
105
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
106 def _check_build_queue(self, when):
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
107 if not self.slaves:
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
108 return
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
109 log.debug('Checking for pending builds...')
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
110 for build in Build.select(self.env, status=Build.PENDING):
86
110bfa3cbc32 Slaves were getting associated with the wrong target platform.
cmlenz
parents: 85
diff changeset
111 for slave in self.slaves.get(build.platform, []):
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
112 active_builds = Build.select(self.env, slave=slave.name,
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
113 status=Build.IN_PROGRESS)
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
114 if not list(active_builds):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
115 slave.send_initiation(build)
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
116 return
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
117
109
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
118 def _cleanup_orphaned_builds(self):
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
119 # Remove all pending or in-progress builds
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
120 db = self.env.get_db_cnx()
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
121 for build in Build.select(self.env, status=Build.IN_PROGRESS, db=db):
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
122 build.status = Build.PENDING
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
123 build.update(db=db)
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
124 for build in Build.select(self.env, status=Build.PENDING, db=db):
112
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
125 for step in BuildStep.select(self.env, build=build.id, db=db):
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
126 for log in BuildLog.select(self.env, build=build.id,
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
127 step=step.name, db=db):
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
128 log.delete(db=db)
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
129 step.delete(db=db)
109
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
130 build.delete(db=db)
112
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
131 db.commit()
109
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
132
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
133 def _cleanup_snapshots(self, when):
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
134 log.debug('Checking for unused snapshot archives...')
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
135 for (config, rev, format), path in self.snapshots.items():
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
136 keep = False
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
137 for build in Build.select(self.env, config=config, rev=rev):
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
138 if build.status not in (Build.SUCCESS, Build.FAILURE):
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
139 keep = True
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
140 break
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
141 if not keep:
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
142 log.info('Removing unused snapshot %s', path)
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
143 os.unlink(path)
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
144 del self.snapshots[(config, rev, format)]
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
145
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
146 def get_snapshot(self, build, format):
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
147 snapshot = self.snapshots.get((build.config, build.rev, format))
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
148 if not snapshot:
96
c8c36f34ff5a Change model class pattern:
cmlenz
parents: 95
diff changeset
149 config = BuildConfig.fetch(self.env, build.config)
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
150 snapshot = archive.pack(self.env, path=config.path, rev=build.rev,
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
151 prefix=config.name, format=format)
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
152 log.info('Prepared snapshot archive at %s' % snapshot)
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
153 self.snapshots[(build.config, build.rev, format)] = snapshot
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
154 return snapshot
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
155
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
156 def register(self, handler):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
157 any_match = False
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
158 for config in BuildConfig.select(self.env):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
159 for platform in TargetPlatform.select(self.env, config=config.name):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
160 if not platform.id in self.slaves:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
161 self.slaves[platform.id] = set()
85
acb7b67b8152 Fix matching of slave properties against target platform rules.
cmlenz
parents: 84
diff changeset
162 match = True
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
163 for property, pattern in ifilter(None, platform.rules):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
164 try:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
165 if not re.match(pattern, handler.info.get(property)):
86
110bfa3cbc32 Slaves were getting associated with the wrong target platform.
cmlenz
parents: 85
diff changeset
166 match = False
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
167 break
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
168 except re.error, e:
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
169 log.error('Invalid platform matching pattern "%s"',
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
170 pattern, exc_info=True)
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
171 match = False
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
172 break
84
30d09249de80 Fix var name in slave registration.
cmlenz
parents: 83
diff changeset
173 if match:
96
c8c36f34ff5a Change model class pattern:
cmlenz
parents: 95
diff changeset
174 log.debug('Slave %s matched target platform %s',
c8c36f34ff5a Change model class pattern:
cmlenz
parents: 95
diff changeset
175 handler.name, platform.name)
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
176 self.slaves[platform.id].add(handler)
86
110bfa3cbc32 Slaves were getting associated with the wrong target platform.
cmlenz
parents: 85
diff changeset
177 any_match = True
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
178
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
179 if not any_match:
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
180 log.warning('Slave %s does not match any of the configured target '
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
181 'platforms', handler.name)
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
182 return False
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
183
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
184 self.schedule(self.check_interval * 0.2, self._check_build_queue)
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
185
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
186 log.info('Registered slave "%s"', handler.name)
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
187 return True
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
188
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
189 def unregister(self, handler):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
190 for slaves in self.slaves.values():
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
191 slaves.discard(handler)
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
192
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
193 for build in Build.select(self.env, slave=handler.name,
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
194 status=Build.IN_PROGRESS):
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
195 log.info('Build [%s] of "%s" by %s cancelled', build.rev,
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
196 build.config, handler.name)
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
197 build.slave = None
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
198 build.status = Build.PENDING
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
199 build.started = 0
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
200 build.update()
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
201 break
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
202 log.info('Unregistered slave "%s"', handler.name)
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
203
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
204
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
205 class OrchestrationProfileHandler(beep.ProfileHandler):
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
206 """Handler for communication on the Bitten build orchestration profile from
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
207 the perspective of the build master.
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
208 """
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
209 URI = 'http://bitten.cmlenz.net/beep/orchestration'
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
210
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
211 def handle_connect(self):
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
212 self.master = self.session.listener
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
213 assert self.master
69
b92d7c7d70fd Record build slave properties in database.
cmlenz
parents: 66
diff changeset
214 self.env = self.master.env
b92d7c7d70fd Record build slave properties in database.
cmlenz
parents: 66
diff changeset
215 assert self.env
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
216 self.name = None
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
217 self.info = {}
24
cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels.
cmlenz
parents: 19
diff changeset
218
cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels.
cmlenz
parents: 19
diff changeset
219 def handle_disconnect(self):
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
220 self.master.unregister(self)
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
221
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
222 def handle_msg(self, msgno, payload):
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
223 assert payload.content_type == beep.BEEP_XML
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
224 elem = xmlio.parse(payload.body)
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
225
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
226 if elem.name == 'register':
69
b92d7c7d70fd Record build slave properties in database.
cmlenz
parents: 66
diff changeset
227 self.name = elem.attr['name']
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
228 for child in elem.children():
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
229 if child.name == 'platform':
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
230 self.info[Build.MACHINE] = child.gettext()
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
231 self.info[Build.PROCESSOR] = child.attr.get('processor')
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
232 elif child.name == 'os':
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
233 self.info[Build.OS_NAME] = child.gettext()
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
234 self.info[Build.OS_FAMILY] = child.attr.get('family')
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
235 self.info[Build.OS_VERSION] = child.attr.get('version')
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
236 self.info[Build.IP_ADDRESS] = self.session.addr[0]
14
1733c601d2f8 Refactored the asyncore loop and shutdown procedure into {{{beep.Initiator}}}.
cmlenz
parents: 13
diff changeset
237
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
238 if not self.master.register(self):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
239 xml = xmlio.Element('error', code=550)[
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
240 'Nothing for you to build here, please move along'
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
241 ]
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
242 self.channel.send_err(msgno, beep.Payload(xml))
83
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4.
cmlenz
parents: 82
diff changeset
243 return
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
244
29
2adc3480e4aa Some cleanup and additional docstrings.
cmlenz
parents: 24
diff changeset
245 xml = xmlio.Element('ok')
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
246 self.channel.send_rpy(msgno, beep.Payload(xml))
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
247
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
248 def send_initiation(self, build):
109
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
249 log.info('Initiating build of "%s" on slave %s', build.config,
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
250 self.name)
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
251
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
252 def handle_reply(cmd, msgno, ansno, payload):
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
253 if cmd == 'ERR':
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
254 if payload.content_type == beep.BEEP_XML:
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
255 elem = xmlio.parse(payload.body)
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
256 if elem.name == 'error':
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
257 log.warning('Slave %s refused build request: %s (%d)',
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
258 self.name, elem.gettext(),
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
259 int(elem.attr['code']))
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
260 return
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
261
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
262 elem = xmlio.parse(payload.body)
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
263 assert elem.name == 'proceed'
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
264 type = encoding = None
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
265 for child in elem.children('accept'):
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
266 type, encoding = child.attr['type'], child.attr.get('encoding')
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
267 if (type, encoding) in (('application/tar', 'gzip'),
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
268 ('application/tar', 'bzip2'),
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
269 ('application/tar', None),
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
270 ('application/zip', None)):
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
271 break
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
272 type = None
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
273 if not type:
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
274 xml = xmlio.Element('error', code=550)[
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
275 'None of the accepted archive formats supported'
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
276 ]
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
277 self.channel.send_err(beep.Payload(xml))
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
278 return
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
279 self.send_snapshot(build, type, encoding)
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
280
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
281 xml = xmlio.Element('build', recipe='recipe.xml')
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
282 self.channel.send_msg(beep.Payload(xml), handle_reply=handle_reply)
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
283
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
284 def send_snapshot(self, build, type, encoding):
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
285
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
286 def handle_reply(cmd, msgno, ansno, payload):
51
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}.
cmlenz
parents: 49
diff changeset
287 if cmd == 'ERR':
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
288 assert payload.content_type == beep.BEEP_XML
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
289 elem = xmlio.parse(payload.body)
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
290 if elem.name == 'error':
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
291 log.warning('Slave %s did not accept archive: %s (%d)',
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
292 self.name, elem.gettext(),
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
293 int(elem.attr['code']))
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
294
109
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
295 elif cmd == 'ANS':
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
296 assert payload.content_type == beep.BEEP_XML
94
e5d1a792aa45 Cleanup and records when a build is aborted.
cmlenz
parents: 93
diff changeset
297 db = self.env.get_db_cnx()
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
298 elem = xmlio.parse(payload.body)
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
299 if elem.name == 'started':
109
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
300 self._build_started(db, build, elem)
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
301 elif elem.name == 'step':
109
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
302 self._build_step_completed(db, build, elem)
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
303 elif elem.name == 'completed':
109
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
304 self._build_completed(db, build, elem)
70
ccd03b6f04ef Updated DTD for orchestration profile.
cmlenz
parents: 69
diff changeset
305 elif elem.name == 'aborted':
109
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
306 self._build_aborted(db, build, elem)
63
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
307 elif elem.name == 'error':
2332aedba328 * Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents: 61
diff changeset
308 build.status = Build.FAILURE
94
e5d1a792aa45 Cleanup and records when a build is aborted.
cmlenz
parents: 93
diff changeset
309 build.update(db=db)
109
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
310 db.commit()
80
dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed.
cmlenz
parents: 76
diff changeset
311
92
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
312 snapshot_format = {
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
313 ('application/tar', 'bzip2'): 'bzip2',
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
314 ('application/tar', 'gzip'): 'gzip',
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
315 ('application/tar', None): 'tar',
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
316 ('application/zip', None): 'zip',
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
317 }[(type, encoding)]
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20.
cmlenz
parents: 90
diff changeset
318 snapshot_path = self.master.get_snapshot(build, snapshot_format)
47
083e848088ee * Improvements to the model classes, and a couple of unit tests.
cmlenz
parents: 45
diff changeset
319 snapshot_name = os.path.basename(snapshot_path)
90
2c4e104afef8 The build master now transmits snapshot archives without blocking while reading the file and sending the BEEP frames. Closes #17.
cmlenz
parents: 88
diff changeset
320 message = beep.Payload(file(snapshot_path), content_type=type,
88
8ae753a22494 Some refactoring of the BEEP implementation:
cmlenz
parents: 86
diff changeset
321 content_disposition=snapshot_name,
90
2c4e104afef8 The build master now transmits snapshot archives without blocking while reading the file and sending the BEEP frames. Closes #17.
cmlenz
parents: 88
diff changeset
322 content_encoding=encoding)
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
323 self.channel.send_msg(message, handle_reply=handle_reply)
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
324
109
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
325 def _build_started(self, db, build, elem):
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
326 build.slave = self.name
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
327 build.slave_info.update(self.info)
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
328 build.started = int(_parse_iso_datetime(elem.attr['time']))
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
329 build.status = Build.IN_PROGRESS
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
330 log.info('Slave %s started build %d ("%s" as of [%s])',
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
331 self.name, build.id, build.config, build.rev)
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
332
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
333 def _build_step_completed(self, db, build, elem):
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
334 log.debug('Slave completed step "%s"', elem.attr['id'])
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
335 step = BuildStep(self.env, build=build.id, name=elem.attr['id'],
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
336 description=elem.attr.get('description'))
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
337 step.started = int(_parse_iso_datetime(elem.attr['time']))
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
338 step.stopped = step.started + int(elem.attr['duration'])
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
339 if elem.attr['result'] == 'failure':
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
340 log.warning('Step failed: %s', elem.gettext())
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
341 step.status = BuildStep.FAILURE
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
342 else:
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
343 step.status = BuildStep.SUCCESS
112
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
344 step.insert(db=db)
109
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
345
112
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
346 # TODO: Store reports, too
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
347 level_map = {'debug': BuildLog.DEBUG, 'info': BuildLog.INFO,
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
348 'warning': BuildLog.WARNING, 'error': BuildLog.ERROR}
109
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
349 for log_elem in elem.children('log'):
112
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
350 build_log = BuildLog(self.env, build=build.id, step=step.name,
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
351 type=log_elem.attr.get('type'))
109
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
352 for messages_elem in log_elem.children('messages'):
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
353 for message_elem in messages_elem.children('message'):
112
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
354 build_log.messages.append((message_elem.attr['level'],
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
355 message_elem.gettext()))
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
356 build_log.insert(db=db)
109
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
357
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
358 def _build_completed(self, db, build, elem):
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
359 log.info('Slave %s completed build %d ("%s" as of [%s])', self.name,
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
360 build.id, build.config, build.rev)
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
361 build.stopped = int(_parse_iso_datetime(elem.attr['time']))
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
362 if elem.attr['result'] == 'failure':
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
363 build.status = Build.FAILURE
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
364 else:
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
365 build.status = Build.SUCCESS
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
366
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
367 def _build_aborted(self, db, build, elem):
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
368 log.info('Slave "%s" aborted build %d ("%s" as of [%s])',
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
369 self.name, build.id, build.config, build.rev)
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
370 build.slave = None
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
371 build.started = 0
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
372 build.status = Build.PENDING
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
373 build.slave_info = {}
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
374 for step in BuildStep.select(self.env, build=build.id, db=db):
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
375 step.delete(db=db)
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents: 96
diff changeset
376
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
377
82
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
378 def _parse_iso_datetime(string):
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
379 """Minimal parser for ISO date-time strings.
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
380
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
381 Return the time as floating point number. Only handles UTC timestamps
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
382 without time zone information."""
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
383 try:
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
384 string = string.split('.', 1)[0] # strip out microseconds
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
385 secs = time.mktime(time.strptime(string, '%Y-%m-%dT%H:%M:%S'))
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
386 tzoffset = time.timezone
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
387 if time.daylight:
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
388 tzoffset = time.altzone
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
389 return secs - tzoffset
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
390 except ValueError, e:
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
391 raise ValueError, 'Invalid ISO date/time %s (%s)' % (string, e)
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents: 80
diff changeset
392
31
0b82d012e42c Add thin script shells around master and slave.
cmlenz
parents: 29
diff changeset
393 def main():
56
033366d81def Build slave now executes the build. Closes #10.
cmlenz
parents: 51
diff changeset
394 from bitten import __version__ as VERSION
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
395 from optparse import OptionParser
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
396
112
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents: 110
diff changeset
397 # Parse command-line arguments
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
398 parser = OptionParser(usage='usage: %prog [options] env-path',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
399 version='%%prog %s' % VERSION)
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
400 parser.add_option('-p', '--port', action='store', type='int', dest='port',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
401 help='port number to use')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
402 parser.add_option('-H', '--host', action='store', dest='host',
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
403 metavar='HOSTNAME',
33
d8d44216258a Exit the slave script when the master disconnects; and other minor fixes.
cmlenz
parents: 31
diff changeset
404 help='the host name or IP address to bind to')
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
405 parser.add_option('-l', '--log', dest='logfile', metavar='FILENAME',
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
406 help='write log messages to FILENAME')
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
407 parser.add_option('-i', '--interval', dest='interval', metavar='SECONDS',
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
408 default=DEFAULT_CHECK_INTERVAL, type='int',
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
409 help='poll interval for changeset detection')
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
410 parser.add_option('--debug', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
411 const=logging.DEBUG, help='enable debugging output')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
412 parser.add_option('-v', '--verbose', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
413 const=logging.INFO, help='print as much as possible')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
414 parser.add_option('-q', '--quiet', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
415 const=logging.ERROR, help='print as little as possible')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
416 parser.set_defaults(port=7633, loglevel=logging.WARNING)
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
417 options, args = parser.parse_args()
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
418
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
419 if len(args) < 1:
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
420 parser.error('incorrect number of arguments')
18
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa)
cmlenz
parents: 15
diff changeset
421 env_path = args[0]
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
422
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
423 # Configure logging
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
424 log = logging.getLogger('bitten')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
425 log.setLevel(options.loglevel)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
426 handler = logging.StreamHandler()
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
427 if options.logfile:
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
428 handler.setLevel(logging.WARNING)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
429 else:
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
430 handler.setLevel(options.loglevel)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
431 formatter = logging.Formatter('%(message)s')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
432 handler.setFormatter(formatter)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
433 log.addHandler(handler)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
434 if options.logfile:
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
435 handler = logging.FileHandler(options.logfile)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
436 handler.setLevel(options.loglevel)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
437 formatter = logging.Formatter('%(asctime)s [%(name)s] %(levelname)s: '
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
438 '%(message)s')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
439 handler.setFormatter(formatter)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
440 log.addHandler(handler)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
441
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
442 port = options.port
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
443 if not (1 <= port <= 65535):
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
444 parser.error('port must be an integer in the range 1-65535')
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
445
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
446 host = options.host
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
447 if not host:
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
448 import socket
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
449 ip = socket.gethostbyname(socket.gethostname())
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
450 try:
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
451 host = socket.gethostbyaddr(ip)[0]
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
452 except socket.error, e:
93
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents: 92
diff changeset
453 log.warning('Reverse host name lookup failed (%s)', e)
19
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
454 host = ip
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents: 18
diff changeset
455
95
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds.
cmlenz
parents: 94
diff changeset
456 master = Master(env_path, host, port, check_interval=options.interval)
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
457 try:
42
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents: 34
diff changeset
458 master.run(timeout=5.0)
13
21aa17f97522 Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff changeset
459 except KeyboardInterrupt:
34
6da9468a6879 The build master now gracefully exits by first terminating all active sessions. Fixes #7.
cmlenz
parents: 33
diff changeset
460 master.quit()
31
0b82d012e42c Add thin script shells around master and slave.
cmlenz
parents: 29
diff changeset
461
0b82d012e42c Add thin script shells around master and slave.
cmlenz
parents: 29
diff changeset
462 if __name__ == '__main__':
33
d8d44216258a Exit the slave script when the master disconnects; and other minor fixes.
cmlenz
parents: 31
diff changeset
463 main()
Copyright (C) 2012-2017 Edgewall Software