Skip to content

Commit f275b22

Browse files
author
A. Jesse Jiryu Davis
committed
Implement read preferences for distributing reads among replica set members PYTHON-367
Replace the 'mongo' dict with a Member object everywhere in ReplicaSetConnection. A handful of commands obey read preferences; most are always sent to primary. Track a 5-sample moving average of each replica set member's ping time. Connection detects whether it's connected to primary, secondary, or mongos.
1 parent 70bf190 commit f275b22

27 files changed

+1619
-386
lines changed

‎doc/api/pymongo/collection.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@
2222
.. autoattribute:: database
2323
.. autoattribute:: slave_okay
2424
.. autoattribute:: read_preference
25+
.. autoattribute:: tag_sets
26+
.. autoattribute:: secondary_acceptable_latency_ms
2527
.. autoattribute:: safe
2628
.. autoattribute:: uuid_subtype
2729
.. automethod:: get_lasterror_options

‎doc/api/pymongo/connection.rst‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,11 +17,15 @@
1717

1818
.. autoattribute:: host
1919
.. autoattribute:: port
20+
.. autoattribute:: is_primary
21+
.. autoattribute:: is_mongos
2022
.. autoattribute:: nodes
2123
.. autoattribute:: max_pool_size
2224
.. autoattribute:: document_class
2325
.. autoattribute:: tz_aware
2426
.. autoattribute:: read_preference
27+
.. autoattribute:: tag_sets
28+
.. autoattribute:: secondary_acceptable_latency_ms
2529
.. autoattribute:: slave_okay
2630
.. autoattribute:: safe
2731
.. autoattribute:: is_locked

‎doc/api/pymongo/database.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@
2020

2121
.. autoattribute:: slave_okay
2222
.. autoattribute:: read_preference
23+
.. autoattribute:: tag_sets
24+
.. autoattribute:: secondary_acceptable_latency_ms
2325
.. autoattribute:: safe
2426
.. automethod:: get_lasterror_options
2527
.. automethod:: set_lasterror_options

‎doc/api/pymongo/index.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313

1414
Alias for :class:`pymongo.replica_set_connection.ReplicaSetConnection`.
1515

16-
.. autoclass:: pymongo.ReadPreference
16+
.. autoclass:: pymongo.read_preferences.ReadPreference
1717
.. autofunction:: has_c
1818

1919
Sub-modules:

‎doc/api/pymongo/replica_set_connection.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@
2121
.. autoattribute:: primary
2222
.. autoattribute:: secondaries
2323
.. autoattribute:: read_preference
24+
.. autoattribute:: tag_sets
25+
.. autoattribute:: secondary_acceptable_latency_ms
2426
.. autoattribute:: max_pool_size
2527
.. autoattribute:: document_class
2628
.. autoattribute:: tz_aware

‎doc/changelog.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,7 @@ Important New Features:
100100
automatic failover handling and periodically checks the state of the
101101
replica set to handle issues like primary stepdown or secondaries
102102
being removed for backup operations. Read preferences are defined through
103-
:class:`~pymongo.ReadPreference`.
103+
:class:`~pymongo.read_preferences.ReadPreference`.
104104
- PyMongo supports the new BSON binary subtype 4 for UUIDs. The default
105105
subtype to use can be set through
106106
:attr:`~pymongo.collection.Collection.uuid_subtype`

‎doc/examples/gevent.rst‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -53,8 +53,9 @@ Additionally, it will use a background greenlet instead of a background thread
5353
to monitor the state of the replica set.
5454

5555
Using :meth:`~pymongo.replica_set_connection.ReplicaSetConnection.start_request()`
56-
with :class:`~pymongo.ReadPreference` PRIMARY ensures that the current greenlet
57-
uses the same socket for all operations until a call to :meth:`end_request()`.
56+
with :class:`~pymongo.read_preferences.ReadPreference` PRIMARY ensures that the
57+
current greenlet uses the same socket for all operations until a call to
58+
:meth:`end_request()`.
5859

5960
You must `install Gevent <http://gevent.org/>`_ to use
6061
:class:`~pymongo.replica_set_connection.ReplicaSetConnection`

‎doc/examples/replica_set.rst‎

Lines changed: 95 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -155,31 +155,113 @@ the operation will succeed::
155155
ReplicaSetConnection
156156
--------------------
157157

158-
In Pymongo-2.1 a new ReplicaSetConnection class was added that provides
159-
some new features not supported in the original Connection class. The most
160-
important of these is the ability to distribute queries to the secondary
161-
members of a replica set. To connect using ReplicaSetConnection just
162-
provide a host:port pair and the name of the replica set::
158+
Using a :class:`~pymongo.replica_set_connection.ReplicaSetConnection` instead
159+
of a simple :class:`~pymongo.connection.Connection` offers two key features:
160+
secondary reads and replica set health monitoring. To connect using
161+
`ReplicaSetConnection` just provide a host:port pair and the name of the
162+
replica set::
163163

164164
>>> from pymongo import ReplicaSetConnection
165165
>>> ReplicaSetConnection("morton.local:27017", replicaSet='foo')
166166
ReplicaSetConnection([u'morton.local:27019', u'morton.local:27017', u'morton.local:27018'])
167167

168+
Secondary Reads
169+
'''''''''''''''
170+
168171
By default an instance of ReplicaSetConnection will only send queries to
169-
the primary member of the replica set. To use secondary members for queries
170-
we have to change the read preference::
172+
the primary member of the replica set. To use secondaries for queries
173+
we have to change the :class:`~pymongo.read_preference.ReadPreference`::
171174

172175
>>> db = ReplicaSetConnection("morton.local:27017", replicaSet='foo').test
173-
>>> from pymongo import ReadPreference
174-
>>> db.read_preference = ReadPreference.SECONDARY
176+
>>> from pymongo.read_preference import ReadPreference
177+
>>> db.read_preference = ReadPreference.SECONDARY_PREFERRED
175178

176179
Now all queries will be sent to the secondary members of the set. If there are
177180
no secondary members the primary will be used as a fallback. If you have
178181
queries you would prefer to never send to the primary you can specify that
179-
using the SECONDARY_ONLY read preference::
182+
using the ``SECONDARY`` read preference::
180183

181-
>>> db.read_preference = ReadPreference.SECONDARY_ONLY
184+
>>> db.read_preference = ReadPreference.SECONDARY
182185

183186
Read preference can be set on a connection, database, collection, or on a
184-
per-query basis.
185-
187+
per-query basis, e.g.::
188+
189+
>>> db.collection.find_one(read_preference=ReadPreference.PRIMARY)
190+
191+
Reads are configured using three options: **read_preference**, **tag_sets**,
192+
and **secondary_acceptable_latency_ms**.
193+
194+
**read_preference**:
195+
196+
- ``PRIMARY``: Read from the primary. This is the default, and provides the
197+
strongest consistency. If no primary is available, raise
198+
:class:`~pymongo.errors.AutoReconnect`.
199+
200+
- ``PRIMARY_PREFERRED``: Read from the primary if available, or if there is
201+
none, read from a secondary matching your choice of ``tag_sets`` and
202+
``secondary_acceptable_latency_ms``.
203+
204+
- ``SECONDARY``: Read from a secondary matching your choice of ``tag_sets`` and
205+
``secondary_acceptable_latency_ms``. If no matching secondary is available,
206+
raise :class:`~pymongo.errors.AutoReconnect`.
207+
208+
- ``SECONDARY_PREFERRED``: Read from a secondary matching your choice of
209+
``tag_sets`` and ``secondary_acceptable_latency_ms`` if available, otherwise
210+
from primary (regardless of the primary's tags and latency).
211+
212+
- ``NEAREST``: Read from any member matching your choice of ``tag_sets`` and
213+
``secondary_acceptable_latency_ms``.
214+
215+
**tag_sets**:
216+
217+
Replica-set members can be `tagged
218+
<http://www.mongodb.org/display/DOCS/Data+Center+Awareness>`_ according to any
219+
criteria you choose. By default, ReplicaSetConnection ignores tags when
220+
choosing a member to read from, but it can be configured with the ``tag_sets``
221+
parameter. ``tag_sets`` must be a list of dictionaries, each dict providing tag
222+
values that the replica set member must match. ReplicaSetConnection tries each
223+
set of tags in turn until it finds a set of tags with at least one matching
224+
member. For example, to prefer reads from the New York data center, but fall
225+
back to the San Francisco data center, tag your replica set members according
226+
to their location and create a ReplicaSetConnection like so:
227+
228+
>>> rsc = ReplicaSetConnection(
229+
... "morton.local:27017",
230+
... replicaSet='foo'
231+
... read_preference=ReadPreference.SECONDARY,
232+
... tag_sets=[{'dc': 'ny'}, {'dc': 'sf'}]
233+
... )
234+
235+
ReplicaSetConnection tries to find secondaries in New York, then San Francisco,
236+
and raises :class:`~pymongo.errors.AutoReconnect` if none are available. As an
237+
additional fallback, specify a final, empty tag set, ``{}``, which means "read
238+
from any member that matches the mode, ignoring tags."
239+
240+
**secondary_acceptable_latency_ms**:
241+
242+
If multiple members match the mode and tag sets, ReplicaSetConnection reads
243+
from among the nearest members, chosen according to ping time. By default,
244+
only members whose ping times are within 15 milliseconds of the nearest
245+
are used for queries. You can choose to distribute reads among members with
246+
higher latencies by setting ``secondary_acceptable_latency_ms`` to a larger
247+
number. In that case, ReplicaSetConnection distributes reads among matching
248+
members within ``secondary_acceptable_latency_ms`` of the closest member's
249+
ping time.
250+
251+
Health Monitoring
252+
'''''''''''''''''
253+
254+
When ReplicaSetConnection is initialized it launches a background task to
255+
monitor the replica set for changes in:
256+
257+
* Health: detect when a member goes down or comes up, or if a different member
258+
becomes primary
259+
* Configuration: detect changes in tags
260+
* Latency: track a moving average of each member's ping time
261+
262+
Replica-set monitoring ensures queries are continually routed to the proper
263+
members as the state of the replica set changes.
264+
265+
It is critical to call
266+
:meth:`~pymongo.replica_set_connection.ReplicaSetConnection.close` to terminate
267+
the monitoring task before your process exits.

‎gridfs/__init__.py‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,8 @@ def __init__(self, database, collection="fs"):
5454
self.__collection = database[collection]
5555
self.__files = self.__collection.files
5656
self.__chunks = self.__collection.chunks
57-
if not database.slave_okay and not database.read_preference:
57+
connection = database.connection
58+
if not hasattr(connection, 'is_primary') or connection.is_primary:
5859
self.__chunks.ensure_index([("files_id", ASCENDING),
5960
("n", ASCENDING)],
6061
unique=True)
@@ -158,7 +159,7 @@ def get_version(self, filename=None, version=-1, **kwargs):
158159
159160
:Parameters:
160161
- `filename`: ``"filename"`` of the file to get, or `None`
161-
- `version` (optional): version of the file to get (defualts
162+
- `version` (optional): version of the file to get (defaults
162163
to -1, the most recent version uploaded)
163164
- `**kwargs` (optional): find files by custom metadata.
164165
@@ -168,8 +169,8 @@ def get_version(self, filename=None, version=-1, **kwargs):
168169
Accept keyword arguments to find files by custom metadata.
169170
.. versionadded:: 1.9
170171
"""
171-
database = self.__database
172-
if not database.slave_okay and not database.read_preference:
172+
connection = self.__database.connection
173+
if not hasattr(connection, 'is_primary') or connection.is_primary:
173174
self.__files.ensure_index([("filename", ASCENDING),
174175
("uploadDate", DESCENDING)])
175176

‎pymongo/__init__.py‎

Lines changed: 1 addition & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -48,49 +48,6 @@
4848
ALL = 2
4949
"""Profile all operations."""
5050

51-
class ReadPreference:
52-
"""An enum that defines the read preferences supported by PyMongo.
53-
54-
+----------------------+--------------------------------------------------+
55-
| Connection type | Read Preference |
56-
+======================+================+================+================+
57-
| |`PRIMARY` |`SECONDARY` |`SECONDARY_ONLY`|
58-
+----------------------+----------------+----------------+----------------+
59-
|Connection to a single|Queries are |Queries are |Same as |
60-
|host. |allowed if the |allowed if the |`SECONDARY` |
61-
| |connection is to|connection is to| |
62-
| |the replica set |the replica set | |
63-
| |primary. |primary or a | |
64-
| | |secondary. | |
65-
+----------------------+----------------+----------------+----------------+
66-
|Connection to a |Queries are sent|Queries are |Same as |
67-
|mongos. |to the primary |distributed |`SECONDARY` |
68-
| |of a shard. |among shard | |
69-
| | |secondaries. | |
70-
| | |Queries are sent| |
71-
| | |to the primary | |
72-
| | |if no | |
73-
| | |secondaries are | |
74-
| | |available. | |
75-
| | | | |
76-
+----------------------+----------------+----------------+----------------+
77-
|ReplicaSetConnection |Queries are sent|Queries are |Queries are |
78-
| |to the primary |distributed |never sent to |
79-
| |of the replica |among replica |the replica set |
80-
| |set. |set secondaries.|primary. An |
81-
| | |Queries are sent|exception is |
82-
| | |to the primary |raised if no |
83-
| | |if no |secondary is |
84-
| | |secondaries are |available. |
85-
| | |available. | |
86-
| | | | |
87-
+----------------------+----------------+----------------+----------------+
88-
"""
89-
90-
PRIMARY = 0
91-
SECONDARY = 1
92-
SECONDARY_ONLY = 2
93-
9451
version_tuple = (2, 2, 1, '+')
9552

9653
def get_version_string():
@@ -103,6 +60,7 @@ def get_version_string():
10360

10461
from pymongo.connection import Connection
10562
from pymongo.replica_set_connection import ReplicaSetConnection
63+
from pymongo.read_preferences import ReadPreference
10664

10765
def has_c():
10866
"""Is the C extension installed?

0 commit comments

Comments
 (0)