Skip to content

Pub / Sub Publisher: Async publish breaks around 1000 messages #4575

@anorth2

Description

@anorth2

Am unsure if this is a bug (can't find documentation explaining this behavior).
I can send up to 999 pubsub messages and wait for results before continuing, but if I try to send 1000, none of them complete. However, if I run in debug mode, they will complete (because I have a breakpoint in the code allowing more execution time).

It exhibits this failure both in the main thread, or calling Publisher.publish from a thread.

  1. OS type and version
    MacOS High Sierra

  2. Python version and virtual environment information python --version
    Python 3.6.3

  3. google-cloud-python version pip show google-cloud, pip show google-<service> or pip freeze

google-api-core==0.1.2
google-auth==1.2.1
google-cloud-core==0.28.0
google-cloud-pubsub==0.29.2
google-gax==0.15.16
googleapis-common-protos==1.5.3
grpc-google-iam-v1==0.11.4

  1. Stacktrace if available

None, hanging, no exception.

  1. Steps to reproduce

Publish 1000 messages storing their futures. None of the futures will complete.

  1. Code example
    Goal here is to send specified number of messages per second and sleep.
    def publish_function(self):
        # save all futures so we can wait for them to complete
        # Callbacks receive the future as their only argument, as defined in
        # the Future interface.
        def callback(future):
            message_id = future.result()
            self.logger.debug(f"Sent message: {message_id}")

        # The callback is added once you get the future. If you add a callback
        # and the future is already done, it will simply be executed immediately.
        future = self.publisher.publish(self.topic_path, data=self.message)
        print(future)
        future.add_done_callback(callback)
        self.results.append(future)

    def publish(self, total_messages, messages_per_second):
        i = 0
        sleep_time = 1
        if messages_per_second < 1:
            sleep_time = float(sleep_time) / float(messages_per_second)
        logger.debug(f"Sleep time is: {sleep_time}")
        while i < total_messages and not self.exit:
            start = time.time()
            # number of times to send per second
            # if less than one, then the sleep timer handles the wait. We just send one message here
            for j in range(int(math.ceil(messages_per_second))):
                self.publish_function()
                i += 1
                self.next_id()
                # exit inner loop if we hit the total to send
                if i >= total_messages or self.exit:
                    self.stop()
                    break

            # apply result to all futures in self.results. This blocks until the future it's called on completes
            # list forces the map to execute since map returns an iterable


            list(map(futures.Future.result, self.results)) <<<<<<<<<<<--------- ***HANGS HERE***

            end = time.time()
            logger.info(
                f"Took {end - start} to send {messages_per_second} messages in thread {current_thread().getName()}")
            logger.debug(f"If greater than 0 seconds, sleeping for {sleep_time - (end - start)}")
            if sleep_time - (end - start) > 0:
                time.sleep(sleep_time - (end - start))

Metadata

Metadata

Assignees

Labels

api: pubsubIssues related to the Pub/Sub API.priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions