Mercurial > babel > mirror

annotate babel/localedata.py @ 249:51a1f6101fa6 trunk

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
added test cases for correct po/mofile sorting, following up r264
author pjenvey
date Mon, 13 Aug 2007 04:16:16 +0000
parents da97a3138239
children 4eca63af0a12
rev   line source
26
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
1 # -*- coding: utf-8 -*-
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
2 #
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
3 # Copyright (C) 2007 Edgewall Software
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
4 # All rights reserved.
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
5 #
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
6 # This software is licensed as described in the file COPYING, which
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
7 # you should have received as part of this distribution. The terms
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
8 # are also available at http://babel.edgewall.org/wiki/License.
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
9 #
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
10 # This software consists of voluntary contributions made by many
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
11 # individuals. For the exact contribution history, see the revision
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
12 # history and logs, available at http://babel.edgewall.org/log/.
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
13
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
14 """Low-level locale data access.
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
15
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
16 :note: The `Locale` class, which uses this module under the hood, provides a
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
17 more convenient interface for accessing the locale data.
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
18 """
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
19
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
20 import os
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
21 import pickle
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
22 try:
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
23 import threading
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
24 except ImportError:
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
25 import dummy_threading as threading
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
26
42
c3bf81204393 Add new function to module exports.
cmlenz
parents: 41
diff changeset
27 __all__ = ['exists', 'load']
26
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
28 __docformat__ = 'restructuredtext en'
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
29
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
30 _cache = {}
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
31 _cache_lock = threading.RLock()
41
e46b7cd193ee Load locale data lazily to avoid penalizing usage of `Locale` objects when no locale data is actually needed.
cmlenz
parents: 26
diff changeset
32 _dirname = os.path.join(os.path.dirname(__file__), 'localedata')
e46b7cd193ee Load locale data lazily to avoid penalizing usage of `Locale` objects when no locale data is actually needed.
cmlenz
parents: 26
diff changeset
33
e46b7cd193ee Load locale data lazily to avoid penalizing usage of `Locale` objects when no locale data is actually needed.
cmlenz
parents: 26
diff changeset
34 def exists(name):
e46b7cd193ee Load locale data lazily to avoid penalizing usage of `Locale` objects when no locale data is actually needed.
cmlenz
parents: 26
diff changeset
35 """Check whether locale data is available for the given locale.
e46b7cd193ee Load locale data lazily to avoid penalizing usage of `Locale` objects when no locale data is actually needed.
cmlenz
parents: 26
diff changeset
36
e46b7cd193ee Load locale data lazily to avoid penalizing usage of `Locale` objects when no locale data is actually needed.
cmlenz
parents: 26
diff changeset
37 :param name: the locale identifier string
e46b7cd193ee Load locale data lazily to avoid penalizing usage of `Locale` objects when no locale data is actually needed.
cmlenz
parents: 26
diff changeset
38 :return: `True` if the locale data exists, `False` otherwise
e46b7cd193ee Load locale data lazily to avoid penalizing usage of `Locale` objects when no locale data is actually needed.
cmlenz
parents: 26
diff changeset
39 :rtype: `bool`
e46b7cd193ee Load locale data lazily to avoid penalizing usage of `Locale` objects when no locale data is actually needed.
cmlenz
parents: 26
diff changeset
40 """
e46b7cd193ee Load locale data lazily to avoid penalizing usage of `Locale` objects when no locale data is actually needed.
cmlenz
parents: 26
diff changeset
41 if name in _cache:
e46b7cd193ee Load locale data lazily to avoid penalizing usage of `Locale` objects when no locale data is actually needed.
cmlenz
parents: 26
diff changeset
42 return True
e46b7cd193ee Load locale data lazily to avoid penalizing usage of `Locale` objects when no locale data is actually needed.
cmlenz
parents: 26
diff changeset
43 return os.path.exists(os.path.join(_dirname, '%s.dat' % name))
26
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
44
185
6503a227ba93 Add a command-line option that prints out all available locales. Closes #24.
cmlenz
parents: 99
diff changeset
45 def list():
6503a227ba93 Add a command-line option that prints out all available locales. Closes #24.
cmlenz
parents: 99
diff changeset
46 """Return a list of all locale identifiers for which locale data is
6503a227ba93 Add a command-line option that prints out all available locales. Closes #24.
cmlenz
parents: 99
diff changeset
47 available.
6503a227ba93 Add a command-line option that prints out all available locales. Closes #24.
cmlenz
parents: 99
diff changeset
48
6503a227ba93 Add a command-line option that prints out all available locales. Closes #24.
cmlenz
parents: 99
diff changeset
49 :return: a list of locale identifiers (strings)
6503a227ba93 Add a command-line option that prints out all available locales. Closes #24.
cmlenz
parents: 99
diff changeset
50 :rtype: `list`
6503a227ba93 Add a command-line option that prints out all available locales. Closes #24.
cmlenz
parents: 99
diff changeset
51 :since: version 0.8.1
6503a227ba93 Add a command-line option that prints out all available locales. Closes #24.
cmlenz
parents: 99
diff changeset
52 """
6503a227ba93 Add a command-line option that prints out all available locales. Closes #24.
cmlenz
parents: 99
diff changeset
53 return [stem for stem, extension in [
6503a227ba93 Add a command-line option that prints out all available locales. Closes #24.
cmlenz
parents: 99
diff changeset
54 os.path.splitext(filename) for filename in os.listdir(_dirname)
6503a227ba93 Add a command-line option that prints out all available locales. Closes #24.
cmlenz
parents: 99
diff changeset
55 ] if extension == '.dat' and stem != 'root']
6503a227ba93 Add a command-line option that prints out all available locales. Closes #24.
cmlenz
parents: 99
diff changeset
56
26
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
57 def load(name):
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
58 """Load the locale data for the given locale.
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
59
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
60 The locale data is a dictionary that contains much of the data defined by
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
61 the Common Locale Data Repository (CLDR). This data is stored as a
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
62 collection of pickle files inside the ``babel`` package.
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
63
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
64 >>> d = load('en_US')
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
65 >>> d['languages']['sv']
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
66 u'Swedish'
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
67
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
68 Note that the results are cached, and subsequent requests for the same
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
69 locale return the same dictionary:
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
70
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
71 >>> d1 = load('en_US')
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
72 >>> d2 = load('en_US')
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
73 >>> d1 is d2
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
74 True
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
75
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
76 :param name: the locale identifier string (or "root")
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
77 :return: the locale data
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
78 :rtype: `dict`
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
79 :raise `IOError`: if no locale data file is found for the given locale
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
80 identifer, or one of the locales it inherits from
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
81 """
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
82 _cache_lock.acquire()
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
83 try:
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
84 data = _cache.get(name)
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
85 if not data:
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
86 # Load inherited data
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
87 if name == 'root':
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
88 data = {}
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
89 else:
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
90 parts = name.split('_')
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
91 if len(parts) == 1:
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
92 parent = 'root'
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
93 else:
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
94 parent = '_'.join(parts[:-1])
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
95 data = load(parent).copy()
41
e46b7cd193ee Load locale data lazily to avoid penalizing usage of `Locale` objects when no locale data is actually needed.
cmlenz
parents: 26
diff changeset
96 filename = os.path.join(_dirname, '%s.dat' % name)
26
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
97 fileobj = open(filename, 'rb')
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
98 try:
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
99 if name != 'root':
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
100 merge(data, pickle.load(fileobj))
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
101 else:
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
102 data = pickle.load(fileobj)
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
103 _cache[name] = data
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
104 finally:
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
105 fileobj.close()
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
106 return data
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
107 finally:
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
108 _cache_lock.release()
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
109
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
110 def merge(dict1, dict2):
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
111 """Merge the data from `dict2` into the `dict1` dictionary, making copies
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
112 of nested dictionaries.
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
113
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
114 :param dict1: the dictionary to merge into
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
115 :param dict2: the dictionary containing the data that should be merged
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
116 """
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
117 for key, value in dict2.items():
233
da97a3138239 Upgraded to CLDR 1.5 and improved timezone formatting.
cmlenz
parents: 185
diff changeset
118 if value is not None:
26
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
119 if type(value) is dict:
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
120 dict1[key] = dict1.get(key, {}).copy()
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
121 merge(dict1[key], value)
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
122 else:
6041782ea677 * Reduce size of locale data pickles by only storing the data provided by each locale itself, and merging inherited data at runtime.
cmlenz
parents:
diff changeset
123 dict1[key] = value
Copyright (C) 2012-2017 Edgewall Software