CPU usage mismatch or usage over 100%

Problem

CPU percent usage data between New Relic APM and New Relic Servers seems contradictory, or your CPU usage exceeds 100%.

Cause

CPU usage data for New Relic APM is different than CPU usage data for New Relic Servers.

  • In APM, CPU percent usage measures aggregate usage of all instances of an application or service on a given server.
  • In Servers, CPU percent usage measures average usage of all CPUs on a given server.
Example of CPU usage mismatch

This example illustrates a discrepancy between CPU percent usage by New Relic APM and New Relic Servers:

This image compares two different CPU metrics in APM and servers.
APM > Monitoring > (selected app) and Servers > Servers: The CPU metrics in New Relic APM and New Relic Servers have different meanings.
New Relic APM

To view your app's CPU usage: From rpm.newrelic.com, select APM > (select an app) > Monitoring > Overview. The number of instances is listed under each host name (for example, 55 instances).

APM CPU percent usage measures aggregate CPU usage of all instances of your app or service on a given server. This percentage is affected by the number of instances running across cores on the server. Multiple instances of a service running on one server or in a multi-core environment can produce CPU usage percentages well above 100%.

APM calculates the percentage by aggregating CPU time and dividing by clock time:

CPU usage = (instance CPU time + instance CPU time + [...]) / (clock time)

Example: Upgrading to a quad core processor

If you upgrade from a dual processor to a quad processor under the same architecture, you should see roughly the same CPU numbers for the same loads and applications. If New Relic normalized the calculation, the upgrade would appear to produce an abrupt decrease in your CPU usage, even if the number of cycles you are using would be the same. Adding more instances does not make your code more efficient.

New Relic Servers

To view your individual server CPU usage:

  1. From rpm.newrelic.com, select Servers.
  2. Optional: To sort the New Relic Servers index by highest or lowest CPU usage, select the CPU column heading.
  3. To drill down into detailed CPU usage information, select the individual host.

The exact calculation for CPU usage depends on your platform:

Platform Calculation details
Linux

New Relic Servers for Linux measures average CPU usage for all CPUs on a given host. New Relic Servers will generate a normalized metric based on the average percentage of time each CPU is busy. This means adding cores to the host causes the CPU capacity to increase:

CPU usage = (CPU time + CPU time + [...]) / (clock time * number of CPUs)
Windows

New Relic Servers for Windows uses Windows Management Instrumentation (WMI) to collect usage for all CPUs and their cores as a single metric. This metric is collected once every twenty seconds using the following query:

SELECT PercentPrivilegedTime,PercentUserTime,TimeStamp_Sys100NS FROM Win32_PerfRawData_PerfOS_Processor WHERE Name = '_Total'

New Relic Servers for Windows then calculates usage individually for PercentPrivilegedTime and PercentUserTime:

( (newCPUTime - oldCPUTime) / (newTimeStamp - oldTimeStamp) ) * 100

For more help

Recommendations for learning more: