-
Notifications
You must be signed in to change notification settings - Fork 41
AMQP reconnections don't clean up sync-out event handlers, causing memory leaks #202
Description
Background
I just had a deep dive into some memory issues of our application. It kept growing over time and we couldn't pinpoint what was causing it. Eventually we saw thousands of references of 131 kB to an instance of Mux from amqplib (amounting to ~2.1GB of unnecessary memory). This led led us to feathers-sync, as we use it with the AMQP adapter.
Update 2025-09-25
See my comment below
Issue
The AMQP adapter doesn't properly clean up the registered app.on('sync-out', handlerFn) event handler when a channel is closed. The handlerFn contains a reference to the channel which is never cleaned up, as the event listener stays active even after a connection is closed.
The AMQP adapter uses amqp-connection-manager for automatic reconnections. The setup option of createChannel (related feathers-sync code) is called each time a new connection is made.
feathers-sync sets up an event listener to sync-out to publish the event to the queue within that setup function. However, in the event of a channel closing (due to whatever reason), the event listener is never cleaned up. Upon new events it'll keep executing the now redundant event listener, as the channel can't be published to anymore.
See the below image for an example of the issue. I added some debug logs to illustrate when a channel is reconnected / closed, as well as which events are sent on which pid / channel. For this I also had to make a small change in amqplib to attach a random string to each created channel.
Scenario: I make a change to an item, wait for the event to be sent, and then I restart my AMQP Docker container to trigger a reconnection.
Impact
- Memory leaks
- The
channelreference (and all other references within it) stay in memory
- The
- Unnecessary processing, as it'll keep trying to call
channel.publishon a closed channel, which will just error
Reproduction
- Setup a Docker container with AMQP (or run it locally)
- Run a Feathers app with the AMQP adapter for
feathers-syncusingDEBUG=feathers-sync:amqpfor debug logs - Trigger something that causes events to be sent via
feathers-sync - Observe how only one "Publish success: |true|" debug is logged
- Restart AMQP
- Trigger the same thing again
- Observe how there is now also a "Publish fail: |Channel closed|" log alongside the "Publish success: |true|" log
- Repeat however many times as you want. Each restart of AMQP will add another "Publish fail: |Channel closed|" log
Proposed solution
Unregister the event listener when a channel is closed. From what I can gather, you can listen to close events on the channel itself. The following seems to fix it for me:
- // Publish the received message to the queue
- app.on('sync-out', (data) => {
- try {
- const publishResponse = channel.publish(
- key,
- queue,
- Buffer.from(data)
- )
- debug(`Publish success: |${publishResponse}| APMQ channel`)
- } catch (error) {
- debug(`Publish fail: |${error.message}| APMQ channel`)
- }
- })
+ function publishToQueue(data) {
+ try {
+ const publishResponse = channel.publish(
+ key,
+ queue,
+ Buffer.from(data)
+ )
+ debug(`Publish success: |${publishResponse}| APMQ channel`)
+ } catch (error) {
+ debug(`Publish fail: |${error.message}| APMQ channel`)
+ }
+ }
+ // Publish the received message to the queue
+ app.on('sync-out', publishToQueue)
+ channel.on('close', () => {
+ debug(Channel closed')
+ app.off('sync-out', publishToQueue)
+ })
+ })Side note: all the debug logs in the AMQP adapter have a typo: APMQ should be AMQP