Thread profiler tool

The thread profiler is a low-impact profiling tool that can be used in production to identify bottlenecks in an application. It works by periodically (100ms) capturing the stack trace of each thread for a specified duration. At the end of the specified duration, the stack traces are aggregated to build a tree. The call count in the tree corresponds to the number of times that function was present in the stack traces under the same context.

Although the call tree cannot capture the entire execution, a large enough sample can be a good representation of the application behavior. This provides insights into the "hot" functions of the app where most of the time is spent.

Access to this feature depends on your subscription level.

Supported agents

Currently this feature is available for:

Start the profiler

The thread profiler feature is enabled by default. You also may be able to turn it on or off in your agent configuration file:

When enabled, you can view the thread profiler from the New Relic user interface:

  1. From the New Relic menu bar, select APM > (selected app) > Events > Thread profiler.
  2. Select the host you want to run the profiler on.
  3. Set the duration for the profiling session.
  4. Select Start profiler.

This triggers the agent to start the thread profiler during the next harvest cycle (every one minute) and capture data for the specified duration. New Relic records thread backtraces whether or not they are in a runnable state at the time the sample is taken. Threads that are sleeping or blocked on IO may appear in the call tree.

This is an image of the thread profiler.
APM > (selected app) > Events > Thread profiler: Use this page to define the settings for the thread profiler duration and to view the results.

View profile data

After the profiler finishes running, the agent will report the profile data. The call tree automatically appears on the Thread profiler page. The percentages in the call tree represent the percentage of thread backtrace samples in which each call path appeared during the profiling session.

thread-profile-view.png
APM > (selected app) > Events > Thread profiler > (selected profile): Here is an example of a call tree from a thread profile.

The page color-codes the tree results:

  • Red: Percentages greater than 30%
  • Yellow: Percentages greater than 10%
  • Black: Percentages less than 10%
If you want to... Do this...
Change how the thread profile information appears Select your choices of available options in the Tree settings, and select Refresh tree.
Change how much information appears Select the Expand or Collapse options above the call tree, or select the name or arrow on any line in the call tree.
View summary information about any line in the call tree Mouse over the line.
Email the thread profile results to others Select Share this profile.
Start another session or view a different thread profile Select Back to all profiles.

Agent considerations

Depending on which New Relic agent you use, the thread profiling feature has additional considerations.

.NET-specific notes

When using thread profiling with the .NET agent, be aware of the following.

.NET agent Thread profiler notes
Managed threads only For .NET agents, the thread profiler only captures stack traces on managed threads. It does not capture stack traces on unmanaged threads. If a call to an unmanaged function occurs on a managed thread, the thread profiler will show Native:Function Call in the call tree.
No line numbers A .NET thread profile does not include line numbers in the call tree. The Show line numbers checkbox in the Tree Settings does not have any effect.
Bug with 64-bit v4.0 .NET CLR

There is a bug in the 64-bit version 4.0 .NET Common Language Runtime (CLR) that interferes with the agent's ability to retrieve managed stack traces. If your app experiences this bug, New Relic APM will show empty thread profiles. This bug does not affect 32-bit applications.

The bug is fixed in the CLR releases for .NET 4.5. To verify whether your 64-bit application has the fixed version, look at the full version of the mscorlib.dll in the C:\Windows\Microsoft.NET\Framework64\v4.0.30319 directory. The fix is in versions 4.0.30319.17379 or higher.

Other category only All threads are put in the Other category. The Web Request and Background categories are not supported.
Python-specific notes

When using thread profiling with the Python agent, be aware of the following.

Python agent Thread profiler notes
Co-routine based systems There are limits to capturing details when a co-routine based system is being used, such as gevent or eventlet modes of gunicorn. If creating a new thread, the Python agent will actually create a greenlet instead of a thread profiler background thread.
Greenlets A greenlet can run only when other greenlets explicitly yield control, such as when they block. For example, if the thread sampler does get to run, it will only sample the stack for other greenlets at a point where they are blocked. It will not sample them when they are executing arbitrary code. It can completely miss execution within a greenlet if it never blocked or otherwise yielded to another greenlet.
Time in Python code Time spent in pure Python code that isn't blocking requests will not be picked up, and no information will be recorded or reported. This is because results are misleading when co-routines are used.
Ruby-specific notes

When using thread profiling with the Ruby agent, be aware of the following.

Ruby agent Thread profiler notes
Backtraces The thread profiler depends on the ability to capture thread backtraces from within your Ruby application. For this reason, it requires MRI 1.9.2 or higher (for the Thread#backtrace method).
Resque The Ruby agent does not currently support thread profiles with Resque background jobs. A thread profiling session initiated against Resque will only capture traces from the parent process, not the job processes.
JRuby JRuby support is considered experimental at this time. There are known issues with JRuby's Thread#backtrace implementation that will affect the accuracy of and reliability of backtraces collected under JRuby.

For more help

Additional documentation resources include:

Join the discussion about New Relic APM in the New Relic Online Technical Community! The Technical Community is a public platform to discuss and troubleshoot your New Relic toolset.

If you need additional help, get support at support.newrelic.com.