Troubleshooting large memory usage (Node.js)

Problem

You installed the New Relic Node.js agent, and your Node.js application's memory usage increased.

Solution

There are several possible causes for this memory increase and potential solutions for each.

Upgrade to Node.js agent version 1.18.0 or higher

The Node.js agent version 1.18.0 introduced a mitigation for a Node.js memory leak that can occur during TLS connections.

Node.js Core has an outstanding memory leak involving TLS connections. Clients that specify certificates quickly reveal this leak, including the New Relic agent. Agent versions 1.18.0 or higher mitigate this issue by using the default client certificates where possible. A new log message is printed when the TLS memory leak workaround cannot be used (for example, when using a custom certificate with an HTTPS proxy).

Increase caused by SSL

To reduce memory leaks caused by TLS in Node.js Core, disable SSL.

If you disable SSL, manually set the port to 80.

Increase caused by TLS memory buffer allocation

The first time a Node.js application uses any form of encryption, including SSL and HTTPS, a slab buffer is created. The default size for this buffer is 10 MB.

Applications running in environments where SSL termination on inbound requests occurs in a separate router layer do not normally incur this overhead. Cloud services like Heroku and AWS often operate this way. However, the Node.js agent sends outbound data to New Relic services over HTTPS, and this triggers the allocation of a slab buffer.

Solution:

In some cases, you can reduce the slab buffer below its 10 MB default.

To set the slab buffer size, use tls.SLAB_BUFFER_SIZE.

When using the New Relic agent, do not set the slab buffer size below 128 KB. The slab buffer allocation should not be reduced for apps that communicate with services or clients using SSL, HTTPS, or any other form of cryptography.

Increase caused by cluster worker slab allocations

Node.js provides the Cluster module. This allows for handling requests in parallel by using all processor cores available on a server. However, each cluster worker allocates its own slab buffer for SSL transactions, and keeps its own copy of Node.js agent data. This multiplies the memory overhead by the number of cluster workers used.

The is also true if a server runs multiple Node.js applications at the same time.

Solution:

Some cloud service providers use server environments that state a higher number of processor cores than can actually be used at any given time. Reducing the number of cluster workers or running without Cluster support may decrease memory usage without impacting performance.

Increase caused by log messages stored to disk

Log messages are logged to disk by default. Due to how message data is handled, message objects may be moved into Old-pointer-space for garbage collection. This means the objects stay in memory for a while, even after all references to them are gone. This leads to a larger amount of memory consumed by a process at any given time. Additional processing time is also used for garbage collection.

Solution:

Depending on your version of the Node.js, the agent may default to the trace or info log levels. Decrease logging verbosity to info or warn levels to noticeably decrease memory usage and time spent in garbage collection.

Increase caused by leaked MongoDB cursors

Many database drivers use an abstraction called a cursor. Cursors provide the ability to iterate through the results of queries. For example, the mongodb driver provides cursors when executing find queries.

Cursors live both as objects in the Node.js runtime and as entities in the MongoDB server. When an application has finished using a cursor, it should close it to free up resources in both the server and the client application.

In Node.js, it is possible for a cursor to be garbage collected, freeing resources in the application, without closing the cursor in the server. This may be go unnoticed in the application. However, the New Relic Node.js agent keeps track of open cursors to measure the time spent iterating through results. If your application does not close all the cursors it uses, the agent will continue to track stale cursors and leak memory.

Solution:

Ensure every cursor created in your application is closed by calling cursor.close() after the application finishes processing the results of the query.

Increase caused by agent data storage

The Node.js agent records data for each transaction your app handles. Data is generally grouped by transaction name. The memory used by the agent increases with the number of different transactions recorded in each minute-long harvest cycle.

In addition, a larger amount of data is kept during each transaction, but is eventually discarded when the transaction completes. Memory used by the agent increases with the number of concurrent transactions handled by the application.

Solution:

If agent data storage is identified as the cause of a memory usage increase, this can best be addressed by adding additional memory to your servers, or by switching to a larger cloud server instance.

For more help

Additional documentation resources include:

Join the discussion about Node.js monitoring in the New Relic Online Technical Community! The Technical Community is a public platform to discuss and troubleshoot your New Relic toolset.

If you need additional help, get support at support.newrelic.com.