Mercurial > bitten > bitten-test

annotate bitten/queue.py @ 446:a8c331c2d500

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Attempt at a fix for #165.
author cmlenz
date Thu, 23 Aug 2007 17:59:58 +0000
parents 22d6a7da8777
children cdce97e97cca
rev   line source
379
0df178e07fdb Use UTF-8 as encoding of source files.
cmlenz
parents: 377
diff changeset
1 # -*- coding: utf-8 -*-
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
2 #
408
933105ab516b Update file headers and other stuff pointing to the old home.
cmlenz
parents: 400
diff changeset
3 # Copyright (C) 2007 Edgewall Software
933105ab516b Update file headers and other stuff pointing to the old home.
cmlenz
parents: 400
diff changeset
4 # Copyright (C) 2005-2007 Christopher Lenz <cmlenz@gmx.de>
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
5 # All rights reserved.
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
6 #
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
7 # This software is licensed as described in the file COPYING, which
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
8 # you should have received as part of this distribution. The terms
408
933105ab516b Update file headers and other stuff pointing to the old home.
cmlenz
parents: 400
diff changeset
9 # are also available at http://bitten.edgewall.org/wiki/License.
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
10
313
90422699a594 More and improved docstrings (using epydoc format).
cmlenz
parents: 295
diff changeset
11 """Implements the scheduling of builds for a project.
90422699a594 More and improved docstrings (using epydoc format).
cmlenz
parents: 295
diff changeset
12
90422699a594 More and improved docstrings (using epydoc format).
cmlenz
parents: 295
diff changeset
13 This module provides the functionality for scheduling builds for a specific
90422699a594 More and improved docstrings (using epydoc format).
cmlenz
parents: 295
diff changeset
14 Trac environment. It is used by both the build master and the web interface to
90422699a594 More and improved docstrings (using epydoc format).
cmlenz
parents: 295
diff changeset
15 get the list of required builds (revisions not built yet).
90422699a594 More and improved docstrings (using epydoc format).
cmlenz
parents: 295
diff changeset
16
411
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
17 Furthermore, the `BuildQueue` class is used by the build master to determine
313
90422699a594 More and improved docstrings (using epydoc format).
cmlenz
parents: 295
diff changeset
18 the next pending build, and to match build slaves against configured target
90422699a594 More and improved docstrings (using epydoc format).
cmlenz
parents: 295
diff changeset
19 platforms.
90422699a594 More and improved docstrings (using epydoc format).
cmlenz
parents: 295
diff changeset
20 """
90422699a594 More and improved docstrings (using epydoc format).
cmlenz
parents: 295
diff changeset
21
437
6d5ac24061dc Fix build queue population compatiblity with Trac 0.11, where revision timestamps are `datetime` objects.
cmlenz
parents: 419
diff changeset
22 from datetime import datetime
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
23 from itertools import ifilter
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
24 import logging
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
25 import re
419
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
26 import time
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
27
400
d039348592bb Fix for missing import in [442].
cmlenz
parents: 395
diff changeset
28 from trac.versioncontrol import NoSuchNode
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
29 from bitten.model import BuildConfig, TargetPlatform, Build, BuildStep
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
30
411
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
31 __docformat__ = 'restructuredtext en'
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
32
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
33 log = logging.getLogger('bitten.queue')
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
34
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
35
258
77cdef044d48 * Improve build log formatter performance: now only matches strings using the `path:line` format, and checks the existance of files in the repository when they are encountered. Should fix (or at least improve) #54.
cmlenz
parents: 253
diff changeset
36 def collect_changes(repos, config, db=None):
228
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
37 """Collect all changes for a build configuration that either have already
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
38 been built, or still need to be built.
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
39
411
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
40 This function is a generator that yields ``(platform, rev, build)`` tuples,
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
41 where ``platform`` is a `TargetPlatform` object, ``rev`` is the identifier
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
42 of the changeset, and ``build`` is a `Build` object or `None`.
313
90422699a594 More and improved docstrings (using epydoc format).
cmlenz
parents: 295
diff changeset
43
411
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
44 :param repos: the version control repository
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
45 :param config: the build configuration
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
46 :param db: a database connection (optional)
228
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
47 """
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
48 env = config.env
258
77cdef044d48 * Improve build log formatter performance: now only matches strings using the `path:line` format, and checks the existance of files in the repository when they are encountered. Should fix (or at least improve) #54.
cmlenz
parents: 253
diff changeset
49 if not db:
77cdef044d48 * Improve build log formatter performance: now only matches strings using the `path:line` format, and checks the existance of files in the repository when they are encountered. Should fix (or at least improve) #54.
cmlenz
parents: 253
diff changeset
50 db = env.get_db_cnx()
395
37e30696c0a8 Don't raise an exception when the repository node associated with a build config has been since removed.
cmlenz
parents: 392
diff changeset
51 try:
37e30696c0a8 Don't raise an exception when the repository node associated with a build config has been since removed.
cmlenz
parents: 392
diff changeset
52 node = repos.get_node(config.path)
37e30696c0a8 Don't raise an exception when the repository node associated with a build config has been since removed.
cmlenz
parents: 392
diff changeset
53 except NoSuchNode, e:
37e30696c0a8 Don't raise an exception when the repository node associated with a build config has been since removed.
cmlenz
parents: 392
diff changeset
54 env.log.warn('Node for configuration %r not found', config.name,
37e30696c0a8 Don't raise an exception when the repository node associated with a build config has been since removed.
cmlenz
parents: 392
diff changeset
55 exc_info=True)
37e30696c0a8 Don't raise an exception when the repository node associated with a build config has been since removed.
cmlenz
parents: 392
diff changeset
56 return
228
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
57
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
58 for path, rev, chg in node.get_history():
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
59
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
60 # Don't follow moves/copies
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
61 if path != repos.normalize_path(config.path):
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
62 break
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
63
231
4a9cf87d443f Allow the build queue for a configuration to be limited to a range between `min_rev` and `max_rev`.
cmlenz
parents: 228
diff changeset
64 # Stay within the limits of the build config
4a9cf87d443f Allow the build queue for a configuration to be limited to a range between `min_rev` and `max_rev`.
cmlenz
parents: 228
diff changeset
65 if config.min_rev and repos.rev_older_than(rev, config.min_rev):
4a9cf87d443f Allow the build queue for a configuration to be limited to a range between `min_rev` and `max_rev`.
cmlenz
parents: 228
diff changeset
66 break
4a9cf87d443f Allow the build queue for a configuration to be limited to a range between `min_rev` and `max_rev`.
cmlenz
parents: 228
diff changeset
67 if config.max_rev and repos.rev_older_than(config.max_rev, rev):
4a9cf87d443f Allow the build queue for a configuration to be limited to a range between `min_rev` and `max_rev`.
cmlenz
parents: 228
diff changeset
68 continue
4a9cf87d443f Allow the build queue for a configuration to be limited to a range between `min_rev` and `max_rev`.
cmlenz
parents: 228
diff changeset
69
228
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
70 # Make sure the repository directory isn't empty at this
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
71 # revision
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
72 old_node = repos.get_node(path, rev)
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
73 is_empty = True
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
74 for entry in old_node.get_entries():
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
75 is_empty = False
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
76 break
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
77 if is_empty:
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
78 continue
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
79
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
80 # For every target platform, check whether there's a build
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
81 # of this revision
258
77cdef044d48 * Improve build log formatter performance: now only matches strings using the `path:line` format, and checks the existance of files in the repository when they are encountered. Should fix (or at least improve) #54.
cmlenz
parents: 253
diff changeset
82 for platform in TargetPlatform.select(env, config.name, db=db):
77cdef044d48 * Improve build log formatter performance: now only matches strings using the `path:line` format, and checks the existance of files in the repository when they are encountered. Should fix (or at least improve) #54.
cmlenz
parents: 253
diff changeset
83 builds = list(Build.select(env, config.name, rev, platform.id,
77cdef044d48 * Improve build log formatter performance: now only matches strings using the `path:line` format, and checks the existance of files in the repository when they are encountered. Should fix (or at least improve) #54.
cmlenz
parents: 253
diff changeset
84 db=db))
228
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
85 if builds:
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
86 build = builds[0]
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
87 else:
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
88 build = None
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
89
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
90 yield platform, rev, build
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
91
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
92
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
93 class BuildQueue(object):
228
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
94 """Enapsulates the build queue of an environment.
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
95
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
96 A build queue manages the the registration of build slaves and detection of
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
97 repository revisions that need to be built.
228
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
98 """
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
99
419
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
100 def __init__(self, env, build_all=False, timeout=0):
228
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
101 """Create the build queue.
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
102
411
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
103 :param env: the Trac environment
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
104 :param build_all: whether older revisions should be built
419
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
105 :param timeout: the time in seconds after which an in-progress build
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
106 should be considered orphaned, and reset to pending
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
107 state
228
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
108 """
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
109 self.env = env
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
110 self.log = env.log
348
ef795ebeac00 Updated change log for [milestone:0.5.3].
cmlenz
parents: 318
diff changeset
111 self.build_all = build_all
419
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
112 self.timeout = timeout
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
113
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
114 # Build scheduling
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
115
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
116 def get_build_for_slave(self, name, properties):
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
117 """Check whether one of the pending builds can be built by the build
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
118 slave.
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
119
411
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
120 :param name: the name of the slave
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
121 :type name: `basestring`
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
122 :param properties: the slave configuration
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
123 :type properties: `dict`
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
124 :return: the allocated build, or `None` if no build was found
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
125 :rtype: `Build`
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
126 """
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
127 log.debug('Checking for pending builds...')
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
128
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
129 db = self.env.get_db_cnx()
348
ef795ebeac00 Updated change log for [milestone:0.5.3].
cmlenz
parents: 318
diff changeset
130 repos = self.env.get_repository()
372
67372eba421c Change building order; rather than just build in reverse revision order (which tends to get stuck if you have a bunch of revisions on a single branch), build the newest revision on each branch before you go backwards and fill in the previous revisions.
wbell
parents: 363
diff changeset
131
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
132 self.reset_orphaned_builds()
372
67372eba421c Change building order; rather than just build in reverse revision order (which tends to get stuck if you have a bunch of revisions on a single branch), build the newest revision on each branch before you go backwards and fill in the previous revisions.
wbell
parents: 363
diff changeset
133
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
134 # Iterate through pending builds by descending revision timestamp, to
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
135 # avoid the first configuration/platform getting all the builds
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
136 platforms = [p.id for p in self.match_slave(name, properties)]
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
137 build = None
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
138 builds_to_delete = []
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
139 for build in Build.select(self.env, status=Build.PENDING, db=db):
377
a465b5dbfecf Tabs and line length.
cmlenz
parents: 372
diff changeset
140 if self.should_delete_build(build, repos):
446
a8c331c2d500 Attempt at a fix for #165.
cmlenz
parents: 444
diff changeset
141 self.log.info('Scheduling build of configuration "%s" at '
a8c331c2d500 Attempt at a fix for #165.
cmlenz
parents: 444
diff changeset
142 'revision [%s] on %r for deletion', config.name,
a8c331c2d500 Attempt at a fix for #165.
cmlenz
parents: 444
diff changeset
143 rev, platform.name)
a8c331c2d500 Attempt at a fix for #165.
cmlenz
parents: 444
diff changeset
144 builds_to_delete.append(build)
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
145 elif build.platform in platforms:
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
146 break
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
147 else:
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
148 self.log.debug('No pending builds.')
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
149 return None
372
67372eba421c Change building order; rather than just build in reverse revision order (which tends to get stuck if you have a bunch of revisions on a single branch), build the newest revision on each branch before you go backwards and fill in the previous revisions.
wbell
parents: 363
diff changeset
150
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
151 # delete any obsolete builds
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
152 for build in builds_to_delete:
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
153 build.delete(db=db)
372
67372eba421c Change building order; rather than just build in reverse revision order (which tends to get stuck if you have a bunch of revisions on a single branch), build the newest revision on each branch before you go backwards and fill in the previous revisions.
wbell
parents: 363
diff changeset
154
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
155 if build:
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
156 build.slave = name
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
157 build.slave_info.update(properties)
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
158 build.status = Build.IN_PROGRESS
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
159 build.update(db=db)
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
160
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
161 if build or builds_to_delete:
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
162 db.commit()
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
163
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
164 return build
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
165
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
166 def match_slave(self, name, properties):
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
167 """Match a build slave against available target platforms.
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
168
411
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
169 :param name: the name of the slave
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
170 :type name: `basestring`
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
171 :param properties: the slave configuration
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
172 :type properties: `dict`
a169d2e96463 Use reStructuredText as the API documentation syntax.
cmlenz
parents: 408
diff changeset
173 :return: the list of platforms the slave matched
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
174 """
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
175 platforms = []
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
176
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
177 for config in BuildConfig.select(self.env):
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
178 for platform in TargetPlatform.select(self.env, config=config.name):
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
179 match = True
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
180 for propname, pattern in ifilter(None, platform.rules):
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
181 try:
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
182 propvalue = properties.get(propname)
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
183 if not propvalue or not re.match(pattern, propvalue):
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
184 match = False
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
185 break
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
186 except re.error:
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
187 self.log.error('Invalid platform matching pattern "%s"',
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
188 pattern, exc_info=True)
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
189 match = False
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
190 break
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
191 if match:
443
1c922d944bd2 Minor improvement to logging in the build master when matching slaves against target platforms.
cmlenz
parents: 437
diff changeset
192 self.log.debug('Slave %r matched target platform %r of '
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
193 'build configuration %r', name,
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
194 platform.name, config.name)
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
195 platforms.append(platform)
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
196
443
1c922d944bd2 Minor improvement to logging in the build master when matching slaves against target platforms.
cmlenz
parents: 437
diff changeset
197 if not platforms:
444
22d6a7da8777 Another minor improvement to logging in the build master.
cmlenz
parents: 443
diff changeset
198 self.log.warning('Slave %r matched none of the target platforms',
22d6a7da8777 Another minor improvement to logging in the build master.
cmlenz
parents: 443
diff changeset
199 name)
443
1c922d944bd2 Minor improvement to logging in the build master when matching slaves against target platforms.
cmlenz
parents: 437
diff changeset
200
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
201 return platforms
372
67372eba421c Change building order; rather than just build in reverse revision order (which tends to get stuck if you have a bunch of revisions on a single branch), build the newest revision on each branch before you go backwards and fill in the previous revisions.
wbell
parents: 363
diff changeset
202
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
203 def populate(self):
228
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
204 """Add a build for the next change on each build configuration to the
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
205 queue.
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
206
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
207 The next change is the latest repository check-in for which there isn't
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
208 a corresponding build on each target platform. Repeatedly calling this
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
209 method will eventually result in the entire change history of the build
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
210 configuration being in the build queue.
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
211 """
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
212 repos = self.env.get_repository()
360
5bc81e597352 Fix incompatibility with a change to the versioncontrol layer in Trac 0.10.3
cmlenz
parents: 352
diff changeset
213 if hasattr(repos, 'sync'):
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
214 repos.sync()
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
215
360
5bc81e597352 Fix incompatibility with a change to the versioncontrol layer in Trac 0.10.3
cmlenz
parents: 352
diff changeset
216 db = self.env.get_db_cnx()
5bc81e597352 Fix incompatibility with a change to the versioncontrol layer in Trac 0.10.3
cmlenz
parents: 352
diff changeset
217 builds = []
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
218
360
5bc81e597352 Fix incompatibility with a change to the versioncontrol layer in Trac 0.10.3
cmlenz
parents: 352
diff changeset
219 for config in BuildConfig.select(self.env, db=db):
5bc81e597352 Fix incompatibility with a change to the versioncontrol layer in Trac 0.10.3
cmlenz
parents: 352
diff changeset
220 for platform, rev, build in collect_changes(repos, config, db):
5bc81e597352 Fix incompatibility with a change to the versioncontrol layer in Trac 0.10.3
cmlenz
parents: 352
diff changeset
221 if build is None:
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
222 self.log.info('Enqueuing build of configuration "%s" at '
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
223 'revision [%s] on %s', config.name, rev,
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
224 platform.name)
437
6d5ac24061dc Fix build queue population compatiblity with Trac 0.11, where revision timestamps are `datetime` objects.
cmlenz
parents: 419
diff changeset
225
6d5ac24061dc Fix build queue population compatiblity with Trac 0.11, where revision timestamps are `datetime` objects.
cmlenz
parents: 419
diff changeset
226 rev_time = repos.get_changeset(rev).date
6d5ac24061dc Fix build queue population compatiblity with Trac 0.11, where revision timestamps are `datetime` objects.
cmlenz
parents: 419
diff changeset
227 if isinstance(rev_time, datetime): # Trac>=0.11
6d5ac24061dc Fix build queue population compatiblity with Trac 0.11, where revision timestamps are `datetime` objects.
cmlenz
parents: 419
diff changeset
228 from trac.util.datefmt import to_timestamp
6d5ac24061dc Fix build queue population compatiblity with Trac 0.11, where revision timestamps are `datetime` objects.
cmlenz
parents: 419
diff changeset
229 rev_time = to_timestamp(rev_time)
6d5ac24061dc Fix build queue population compatiblity with Trac 0.11, where revision timestamps are `datetime` objects.
cmlenz
parents: 419
diff changeset
230
360
5bc81e597352 Fix incompatibility with a change to the versioncontrol layer in Trac 0.10.3
cmlenz
parents: 352
diff changeset
231 build = Build(self.env, config=config.name,
5bc81e597352 Fix incompatibility with a change to the versioncontrol layer in Trac 0.10.3
cmlenz
parents: 352
diff changeset
232 platform=platform.id, rev=str(rev),
437
6d5ac24061dc Fix build queue population compatiblity with Trac 0.11, where revision timestamps are `datetime` objects.
cmlenz
parents: 419
diff changeset
233 rev_time=rev_time)
360
5bc81e597352 Fix incompatibility with a change to the versioncontrol layer in Trac 0.10.3
cmlenz
parents: 352
diff changeset
234 builds.append(build)
5bc81e597352 Fix incompatibility with a change to the versioncontrol layer in Trac 0.10.3
cmlenz
parents: 352
diff changeset
235 break
437
6d5ac24061dc Fix build queue population compatiblity with Trac 0.11, where revision timestamps are `datetime` objects.
cmlenz
parents: 419
diff changeset
236
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
237 if not self.build_all:
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
238 self.log.debug('Ignoring older revisions for configuration '
446
a8c331c2d500 Attempt at a fix for #165.
cmlenz
parents: 444
diff changeset
239 '%r on %r', config.name, platform.name)
a8c331c2d500 Attempt at a fix for #165.
cmlenz
parents: 444
diff changeset
240 continue
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
241
360
5bc81e597352 Fix incompatibility with a change to the versioncontrol layer in Trac 0.10.3
cmlenz
parents: 352
diff changeset
242 for build in builds:
5bc81e597352 Fix incompatibility with a change to the versioncontrol layer in Trac 0.10.3
cmlenz
parents: 352
diff changeset
243 build.insert(db=db)
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
244
360
5bc81e597352 Fix incompatibility with a change to the versioncontrol layer in Trac 0.10.3
cmlenz
parents: 352
diff changeset
245 db.commit()
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
246
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
247 def reset_orphaned_builds(self):
419
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
248 """Reset all in-progress builds to ``PENDING`` state if they've been
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
249 running so long that the configured timeout has been reached.
228
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
250
419
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
251 This is used to cleanup after slaves that have unexpectedly cancelled
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
252 a build without notifying the master, or are for some other reason not
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
253 reporting back status updates.
228
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236].
cmlenz
parents: 227
diff changeset
254 """
419
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
255 if not self.timeout:
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
256 # If no timeout is set, none of the in-progress builds can be
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
257 # considered orphaned
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
258 return
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
259
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
260 db = self.env.get_db_cnx()
419
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
261 now = int(time.time())
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
262 for build in Build.select(self.env, status=Build.IN_PROGRESS, db=db):
419
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
263 if now - build.started < self.timeout:
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
264 # This build has not reached the timeout yet, assume it's still
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
265 # being executed
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
266 # FIXME: ideally, we'd base this check on the last activity on
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
267 # the build, not the start time
b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option.
cmlenz
parents: 411
diff changeset
268 continue
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
269 build.status = Build.PENDING
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
270 build.slave = None
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
271 build.slave_info = {}
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
272 build.started = 0
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
273 for step in list(BuildStep.select(self.env, build=build.id, db=db)):
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
274 step.delete(db=db)
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
275 build.update(db=db)
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
276 db.commit()
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
277
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
278 def should_delete_build(self, build, repos):
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
279 # Ignore pending builds for deactived build configs
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
280 config = BuildConfig.fetch(self.env, build.config)
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
281 if not config.active:
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
282 log.info('Dropping build of configuration "%s" at '
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
283 'revision [%s] on "%s" because the configuration is '
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
284 'deactivated', config.name, build.rev,
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
285 TargetPlatform.fetch(self.env, build.platform).name)
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
286 return True
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
287
392
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
288 # Stay within the revision limits of the build config
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
289 if (config.min_rev and repos.rev_older_than(build.rev,
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
290 config.min_rev)) \
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
291 or (config.max_rev and repos.rev_older_than(config.max_rev,
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
292 build.rev)):
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
293 # This minimum and/or maximum revision has changed since
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
294 # this build was enqueued, so drop it
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
295 log.info('Dropping build of configuration "%s" at revision [%s] on '
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
296 '"%s" because it is outside of the revision range of the '
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
297 'configuration', config.name, build.rev,
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
298 TargetPlatform.fetch(self.env, build.platform).name)
026d9aa41b85 Merged HTTP branch into trunk.
cmlenz
parents: 379
diff changeset
299 return True
227
014bc6c29dff * Factor build queue logic into a class separate from the build master.
cmlenz
parents:
diff changeset
300
253
cda723f3ac31 Provide hooks for build notification. Closes #62.
cmlenz
parents: 239
diff changeset
301 return False
Copyright (C) 2012-2017 Edgewall Software