Skip to content

AMQP reconnections don't clean up sync-out event handlers, causing memory leaks #202

@JulienZD

Description

@JulienZD

Background

I just had a deep dive into some memory issues of our application. It kept growing over time and we couldn't pinpoint what was causing it. Eventually we saw thousands of references of 131 kB to an instance of Mux from amqplib (amounting to ~2.1GB of unnecessary memory). This led led us to feathers-sync, as we use it with the AMQP adapter.

Update 2025-09-25

See my comment below

Issue

The AMQP adapter doesn't properly clean up the registered app.on('sync-out', handlerFn) event handler when a channel is closed. The handlerFn contains a reference to the channel which is never cleaned up, as the event listener stays active even after a connection is closed.

The AMQP adapter uses amqp-connection-manager for automatic reconnections. The setup option of createChannel (related feathers-sync code) is called each time a new connection is made.
feathers-sync sets up an event listener to sync-out to publish the event to the queue within that setup function. However, in the event of a channel closing (due to whatever reason), the event listener is never cleaned up. Upon new events it'll keep executing the now redundant event listener, as the channel can't be published to anymore.

See the below image for an example of the issue. I added some debug logs to illustrate when a channel is reconnected / closed, as well as which events are sent on which pid / channel. For this I also had to make a small change in amqplib to attach a random string to each created channel.

Scenario: I make a change to an item, wait for the event to be sent, and then I restart my AMQP Docker container to trigger a reconnection.

Image

Impact

  • Memory leaks
    • The channel reference (and all other references within it) stay in memory
  • Unnecessary processing, as it'll keep trying to call channel.publish on a closed channel, which will just error

Reproduction

  1. Setup a Docker container with AMQP (or run it locally)
  2. Run a Feathers app with the AMQP adapter for feathers-sync using DEBUG=feathers-sync:amqp for debug logs
  3. Trigger something that causes events to be sent via feathers-sync
  4. Observe how only one "Publish success: |true|" debug is logged
  5. Restart AMQP
  6. Trigger the same thing again
  7. Observe how there is now also a "Publish fail: |Channel closed|" log alongside the "Publish success: |true|" log
  8. Repeat however many times as you want. Each restart of AMQP will add another "Publish fail: |Channel closed|" log

Proposed solution

Unregister the event listener when a channel is closed. From what I can gather, you can listen to close events on the channel itself. The following seems to fix it for me:

-          // Publish the received message to the queue
-          app.on('sync-out', (data) => {
-            try {
-              const publishResponse = channel.publish(
-                key,
-                queue,
-                Buffer.from(data)
-              )
-              debug(`Publish success: |${publishResponse}| APMQ channel`)
-           } catch (error) {
-              debug(`Publish fail: |${error.message}| APMQ channel`)
-           }
-          })

+         function publishToQueue(data) {
+           try {
+              const publishResponse = channel.publish(
+                key,
+                queue,
+                Buffer.from(data)
+              )
+              debug(`Publish success: |${publishResponse}| APMQ channel`)
+            } catch (error) {
+              debug(`Publish fail: |${error.message}| APMQ channel`)
+            }
+          }

+          // Publish the received message to the queue
+          app.on('sync-out', publishToQueue)

+          channel.on('close', () => {
+            debug(Channel closed')
+            app.off('sync-out', publishToQueue)
+          })
+        })

Side note: all the debug logs in the AMQP adapter have a typo: APMQ should be AMQP

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions