Runtime Integration Tests: test_connector_status fails with 'Failed to flush all Kafka messages' after topic partition count drops to 0 #7550

Verdict: infrastructure flake, unrelated to any PR's code. Kafka producer failed to flush messages to a freshly-created topic within the test's 30s timeout.

Failing test/job:

• tests/runtime/test_connector_status.py::test_connector_status
• Job: Integration Tests / Runtime Integration Tests (gke-runners-amd64)
• Seen on merge queue run for PR [adapters] Compact merge-mode tables between flushes #7525 (delta-merge-auto-compact), which only touches Delta Lake merge/compact code — nothing in the Kafka connector/test path.

Evidence (log lines from the job, timestamps ~16:47:24):

RuntimeError: Failed to flush all Kafka messages, remaining=500
tests/runtime/test_connector_status.py:24: RuntimeError

and in the teardown stderr:

%4|...|TERMINATE|rdkafka#producer-2| [thrd:app]: Producer terminating with 500 messages (5390 bytes) still in queue or transit: use flush() to wait for outstanding message delivery
%5|...|PARTCNT|rdkafka#producer-2| [thrd:main]: Topic connector-errors-input-34db6dc9dba4 partition count changed from 1 to 0

The partition count dropping to 0 right after the topic was created suggests the Kafka broker/metadata briefly lost or never properly propagated the topic's partitions, so none of the 500 produced records could be delivered inside the 30s flush() timeout.

What to look at next:

• Whether topic creation (create_topic in python/tests/kafka.py) needs to wait for partition metadata propagation before producers start, or whether the Kafka broker container in this CI environment is flaky around topic creation.
• No prior occurrence found in open fred-observe issues; related-but-distinct existing infra flakiness: CI: Runtime Integration Tests fail with k8s ENHANCE_YOUR_CALM (ResourceExhausted) during pipeline ConfigMap creation #7465, Adapter/Runtime Integration Tests: network-timeout flakiness to GCS/S3/Iceberg-REST/Unity (not PR #7393's fault) #7440, arm64 Runtime Integration Tests: OIDC token starts getting 401 'credential rejected' ~40min into the run #7048.

Not a PR bug — requeueing #7525 (or any other PR hitting this) should pass as long as the Kafka container cooperates.