[RFC] Chained message handling in Symfony Messenger #50462
Description
Description
This proposal introduces a feature into Symfony Messenger which allows for messages to be handled in a chained manner, similar to Laravel's job chaining: https://laravel.com/docs/10.x/queues#job-chaining
In Laravel, a user can chain jobs together to ensure they get executed one after another. This is done by dispatching an array of job instances to the Bus::chain() method. The chain is a series of related jobs that need to run in a particular order. If a job in the chain fails, all remaining jobs on the chain will not be processed. Optionally, an error callback can be specified that is invoked when any of the jobs in the chain fail.
Bus::chain([
new ProcessPodcast($podcast),
new OptimizePodcast($podcast),
new ReleasePodcast($podcast),
])
->catch(fn ($error) => $podcast->update(['status' => 'failed']))
->dispatch();Why would you want this?
- Easier to create complex workflows.
- Less coupling between handlers.
- Better error management.
In Symfony, the Messenger component doesn't provide a similar native functionality. The approach I lean towards now is: dispatch a new message at the end of a handler, but this causes the two handlers to be tightly coupled.
Proposal
I've tried to implement this feature myself. I've ended up with this syntax:
$this->messageBus->dispatch(
Chain::make([
new ProcessPodcast($podcastId),
new OptimizePodcast($podcastId),
new ReleasePodcast($podcastId)
])
);Chain::make() takes the first message and adds a ChainStamp, which contains all the subsequent messages:
class Chain
{
public static function make(array $messages): Envelope
{
$firstMessage = Envelope::wrap($messages[0]);
return $firstMessage->with(
new ChainStamp(array_slice($messages, 1))
);
}
}Then there's a middleware called ChainMiddleware which dispatches the next message in the chain after the first message is handled:
class ChainMiddleware implements MiddlewareInterface
{
public function __construct(private readonly MessageBusInterface $messageBus)
{
}
public function handle(Envelope $envelope, StackInterface $stack): Envelope
{
$envelope = $stack->next()->handle($envelope, $stack);
$chainStamp = $envelope->last(ChainStamp::class);
$handledStamp = $envelope->last(HandledStamp::class);
if (!$chainStamp || !$handledStamp || count($chainStamp->getChain()) === 0) {
return $envelope;
}
$this->messageBus->dispatch(
Chain::make($chainStamp->getChain()),
);
return $envelope;
}
}This works. Now on to the error handler, I tried to do something like this:
$this->messageBus->dispatch(
Chain::make([
new ProcessPodcast($podcastId),
new OptimizePodcast($podcastId),
new ReleasePodcast($podcastId)
], onError: fn () => dd('whoops'))
);This particular example is quite easy using opis/closure, but it's not very useful. We need access to dependencies to be able to do something useful: e.g. entity manager to set the status of some entity to failed. Now how do you get access to dependencies for a deserialised closure...? I tried to make this work:
$this->messageBus->dispatch(
Chain::make([
new ProcessPodcast($podcastId),
new OptimizePodcast($podcastId),
new ReleasePodcast($podcastId)
], onError: fn (LoggerInterface $logger) => $logger->info('Hi!'))
);Then I tried to get access to the dependencies using reflection & the service container:
$closure = $serializableClosure->getClosure();
$parameters = (new ReflectionFunction($closure))->getParameters();
$args = [];
// For each parameter, fetch the appropriate service from the container
foreach ($parameters as $parameter) {
$typeHint = $parameter->getType();
$invadedContainer = invade($this->container);
$args[] = $invadedContainer->get($typeHint->getName(), ContainerInterface::NULL_ON_INVALID_REFERENCE)
?? $invadedContainer->privates($typeHint->getName());
}
$closure(...$args);invade() is from spatie/invade to bypass the inability to access private dependencies. Not a fan of this approach, but that's why I made the RFC. To get better ideas from smarter people 😉
A case to keep in mind
There's some difficulties with this approach for the following use case: we often hit rate limits within our handler which communicates with an external API. This API return headers explaining when this API call can be retried. We then use something like this to retry after X seconds:
$this->messageBus->dispatch(
(new Envelope($this))->with(new DelayStamp($response->header('retry-after')))
);This is problematic within a chain because it discards the whole remaining chain. A possible solution here is to to have some kind of RetryAfterException(int $delay) which is picked up by a middleware and then the original envelope is stamped with a DelayStamp and re-dispatched. It's just something I realised when building this.
Extra information
- This RFC was sparked by this thread on Twitter: https://twitter.com/nicolasgrekas/status/1663108014390358016
- Source code of what I tried can be found here: https://github.com/dejagersh/my-messenger-sandbox/
Example
$this->messageBus->dispatch(
Chain::make([
new ProcessPodcast($podcastId),
new OptimizePodcast($podcastId),
new ReleasePodcast($podcastId)
], onError: function (PodcastRepository $podcastRepository) use ($podcastId) {
$podcast = $podcastRepository->find($podcastId);
$podcast->setStatus('failed');
$podcastRepository->save($podcast, flush: true);
})
);