-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Description
Am unsure if this is a bug (can't find documentation explaining this behavior).
I can send up to 999 pubsub messages and wait for results before continuing, but if I try to send 1000, none of them complete. However, if I run in debug mode, they will complete (because I have a breakpoint in the code allowing more execution time).
It exhibits this failure both in the main thread, or calling Publisher.publish from a thread.
-
OS type and version
MacOS High Sierra -
Python version and virtual environment information
python --version
Python 3.6.3 -
google-cloud-python version
pip show google-cloud,pip show google-<service>orpip freeze
google-api-core==0.1.2
google-auth==1.2.1
google-cloud-core==0.28.0
google-cloud-pubsub==0.29.2
google-gax==0.15.16
googleapis-common-protos==1.5.3
grpc-google-iam-v1==0.11.4
- Stacktrace if available
None, hanging, no exception.
- Steps to reproduce
Publish 1000 messages storing their futures. None of the futures will complete.
- Code example
Goal here is to send specified number of messages per second and sleep.
def publish_function(self):
# save all futures so we can wait for them to complete
# Callbacks receive the future as their only argument, as defined in
# the Future interface.
def callback(future):
message_id = future.result()
self.logger.debug(f"Sent message: {message_id}")
# The callback is added once you get the future. If you add a callback
# and the future is already done, it will simply be executed immediately.
future = self.publisher.publish(self.topic_path, data=self.message)
print(future)
future.add_done_callback(callback)
self.results.append(future)
def publish(self, total_messages, messages_per_second):
i = 0
sleep_time = 1
if messages_per_second < 1:
sleep_time = float(sleep_time) / float(messages_per_second)
logger.debug(f"Sleep time is: {sleep_time}")
while i < total_messages and not self.exit:
start = time.time()
# number of times to send per second
# if less than one, then the sleep timer handles the wait. We just send one message here
for j in range(int(math.ceil(messages_per_second))):
self.publish_function()
i += 1
self.next_id()
# exit inner loop if we hit the total to send
if i >= total_messages or self.exit:
self.stop()
break
# apply result to all futures in self.results. This blocks until the future it's called on completes
# list forces the map to execute since map returns an iterable
list(map(futures.Future.result, self.results)) <<<<<<<<<<<--------- ***HANGS HERE***
end = time.time()
logger.info(
f"Took {end - start} to send {messages_per_second} messages in thread {current_thread().getName()}")
logger.debug(f"If greater than 0 seconds, sleeping for {sleep_time - (end - start)}")
if sleep_time - (end - start) > 0:
time.sleep(sleep_time - (end - start))