Reference counting over process boundaries #2056

Hi,

we have the following situation: Actor A is running in some process P1, and is calling a handler in actor B running in process P2. In turn, actor B delegates the request to another actor C running in the same process P2. Actor C's lifetime is managed by reference count, ie. it is expected to shut down automatically when its parent actor is done with its shutdown.

In our setup, we observe the delegation to be working, and process P1 is shutting down correctly after receiving the response.

However, afterwards when we shut down process P2, we observe the process to be hanging, because C never exits. For debugging purposes, we attached a functor to C, which is never called.

The relevant message handler of actor B looks like this: (full source code can be seen here: https://github.com/tenzir/tenzir/pull/5065/files#diff-10866b7c9feebd59bc9edca0a53463326cdf9c1cd42e3a597e2cdf8e99441f8c )

    [self](atom::resolve, std::string name,
           std::string public_key) -> caf::result<secret_resolution_result> {
      // [...]
      auto store = self->system().registry().get("tenzir.platform");
      auto typed_store = caf::actor_cast<secret_store_actor>(store);
      return self->mail(atom::resolve_v, std::move(name), std::move(public_key))
        .delegate(typed_store);
    },

Changing the return statement to a request/response pattern resolves the issue:

      auto rp = self->make_response_promise<secret_resolution_result>();
      self->mail(atom::resolve_v, std::move(name), std::move(public_key))
        .request(typed_store, defaults::secret_lookup_timeout)
        .then(
          [rp = rp](secret_resolution_result r) mutable {
            rp.deliver(std::move(r));
          },
          [rp = rp](caf::error e) mutable {
            rp.deliver(std::move(e));
          });
      return rp;