JMX monitoring integration

New Relic Infrastructure's Java Management Extensions (JMX) integration collects data from any application that reports metrics via JMX. This document explains how to install and configure the JMX integration, and describes the types of metrics collected.

Access to this feature depends on your subscription level. Requires Infrastructure Pro.

Features

The New Relic JMX on-host integration allows users to monitor any application that exposes metrics with JMX. The metrics to be collected are defined in YAML files, which can be configured to collect any subset of metrics desired. A default collection file is included that contains key metrics exposed by the JVM.

Compatibility and requirements

To use the JMX integration, ensure your system meets these requirements:

Install

On-host integrations do not automatically update. For best results, you should occasionally update the integration and update the Infrastructure agent.

To install the JMX integration:

  1. Follow the instructions for installing an integration, using the file name nri-jmx.
  2. Via the command line, change directory to the integrations folder:
    cd /etc/newrelic-infra/integrations.d
  3. Create a copy of the sample configuration file by running:
    sudo cp jmx-config.yml.sample jmx-config.yml
  4. Create a copy of the JVM configuration file by running:
    sudo cp jvm-metrics.yml.sample jvm-metrics.yml
  5. Optional: If you're interested in monitoring Tomcat, we provide a sample metrics file you can use.
    sudo cp tomcat-metrics.yml.sample tomcat-metrics.yml
  6. Edit the jmx-config.yml configuration file using the configuration settings.
  7. Restart the infrastructure agent.

When the infrastructure agent executes the nri-jmx binary, it sets the path to PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin.

The java binary must be in one of those paths.

Configure

The JMX integration defines and collects integration data using two types of YAML files; jmx-config.yml configuration file and metrics collection files.

The jmx-config.yml is where the host connection and collection file information is defined. The file accepts the following arguments. For an example configuration, see the host connection file example.

  • jmx_host: The host JMX is running on. Default: localhost.
  • jmx_port: The port JMX is running on. Default: 9999.
  • jmx_user: The username for the JMX connection. Default: admin.
  • jmx_pass: The password for the JMX connection. Default: admin.
  • collection_files: A comma-separated list of full file paths to the metric collection definition files. Default JVM metrics collection file: /etc/newrelic-infra/integrations.d/jvm-metrics.yml.

The metrics collection definition files are structured YAML files which tell the integration what metrics to collect. For an example configuration, see the metrics collection file example.

Default JVM metrics collection file: /etc/newrelic-infra/integrations.d/jvm-metrics.yml

You can write different collection files to ease organization and maintenance. See configuration file below for an example.

The integration collects and organizes metrics accoriding to domains. All metrics defined per domain will be sent to New Relic and can be found in a corresponding event type. This event type is either auto-generated or can be set by the user. Each file contains a single collect: block which contains an array of domains. For each domain, the following keys are defined:

  • domain: The JMX domain, e.g., java.lang. This can be wildcarded to match multiple domains, e.g., java.*. This field is required.
  • event_type: The event type name for a collection from this domain. If the domain is wildcarded, this is required. If the domain is not wildcarded and this is undefined by the user, this will be auto generated. As an example, the domain java.lang will have event type JavaLangSample.

    For tips on naming event types, see Naming tips.

  • beans: An array of beans to collect in this domain. See Beans.

There is a limit of 200 metrics per instance in the configuration file. If you exceed the limit for a particular instance, it will not be sent to New Relic. If you're not seeing your data in New Relic, please check out the trouble shooting guide below to identify if you have exceeded the limit.

Each domain contains an array of beans to be collected. For each bean, the following keys are defined:

  • query: The bean name to collect, e.g., type=GarbageCollector,name=YoungGen. This can be wildcarded, e.g., type=GarbageCollector,name=*. This field is required.
  • exclude_regex: An optional list of regex patterns that match beans to exclude from collection, e.g., type=GarbageCollector,name=.*.

  • attributes: A list of attributes to collect. If unspecified, collects all attributes. See Attributes.

Each bean can contain attributes, an optional list of beans that can be excluded from collection. For each attribute, the following keys are defined:

For map attributes, you must define either an attr or an attr_regex key.

  • attr: An exact match of the attribute name. Composite attributes can be collected by appending the composite member name to the attribute name with a dot, e.g., HeapMemoryUsage.Max.
  • attr_regex: A regex pattern that matches the attributes to be collected.

  • metric_type: The New Relic metric type to collect this attribute as. Options are:

    • gauge: data will be collected as an instantaneous numeric measurement.
    • rate: data will be collected as the change in that metric per second.
    • delta: data will be collected as the change in that metric since the last measurement.
    • attribute: data will be collected as a string literal.

    If left unspecified, the JMX integration will attempt to infer the metric type based on the value returned. For example, if the metric is a number, it will collect it as gauge. If the metric is a string, it will collect it as attribute.

    If metrics are collected with an incorrect metric type, you can manually specify the correct metric type in the collection file.

  • metric_name: The name under which the metric will appear in New Relic. If unspecified, it will default to the attribute name.

For more information on JMX Queries, please see the Oracle ObjectName documentation.

Example file configurations:

Example host connection file
        integration_name: com.newrelic.jmx

        instances:
          - name: jmx
            command: all_data
            arguments:
                jmx_host: jmx-host.localnet
                jmx_port: 9999
                jmx_user: admin
                jmx_pass: admin
                collection_files: "/etc/newrelic-infra/integrations.d/jvm-metrics.yml,/etc/newrelic-infra/integrations.d/tomcat-metrics.yml"
      
Example metrics collection file
      collect:
            # The event type for this domain will be JavaLangSample
          - domain: java.lang 
            beans:
                # Collect all beans of type Threading
              - query: type=Threading
                # Attributes can be either a string or a map
                attribute:
                      # When unspecified, the metric_type is inferred
                      # and the metric name is just the attribute name
                    - ThreadCount
                      # If using a map attribute, a custom metric name can be set
                    - attr: TotalStartedThreadCount
                      metric_name: ThreadsStarted
                      # Attributes can be collected with regex matches and
                      # the metric type can be overridden if the integration
                      # can not correctly infer the type
                    - attr_regex: "ThreadCpu.*Enabled"
                      metric_type: attribute
              - query: type=Memory 
                attributes:
                      # Composite attributes can be collected with this syntax
                    - HeapMemoryUsage.Max 
                    - NonHeapMemoryUsage.Max 
                # Queries can be wildcarded where
              - query: type=GarbageCollector,name=*
                # If a specific bean is unwanted, it can be excluded
                # with a regex match pattern. Useful if using a wildcard query
                exclude_regex: 
                      # This will match any bean where the name is YoungGen
                    - name=YoungGen
                attributes:
                    - attr: LastGcInfo.GcThreadCount
                      metric_type: gauge
                      metric_name: GCThreadCount
            # Domains can be wildcarded
          - domain: java.util.*
            # If the domain is wildcarded, a custom event must be defined
            event_type: JavaUtilSample 
            beans:
                  # If no attributes are defined, all are collected by default
                - query: type=Logging
        

Tips for naming your data

Metrics are sent and stored in New Relic Infrastructure in the form of samples; that is, a list of key-value pairs that include metric data and metadata. Each sample is stored as an event in New Relic’s event database.

You are responsible for creating and naming the JMX data reported to New Relic. For this reason, New Relic strongly recommends following the convention below when naming your event types to ensure you have a consistent naming scheme:

  • Use camel case.
  • Use a name that clearly identifies what data it contains.

Example: MyorgApplicationSample.

It's also recommended to use the same naming scheme for similar metrics across different applications.

Find and use data

The integration will present data about the JVM in Infrastructure. To find your integration data in Infrastructure, go to infrastructure.newrelic.com > Integrations > On-host integrations and select one of the JMX integration links to view your dashboard. A default dashboard displaying your metrics is included with the JMX integration.

For more on how to find and use your integrations data in Infrastructure, see Understand integration data.

All other metrics can be found in New Relic Insights, JMX data is attached to the user defined event type specified in the configuration file.

For example, if you are interested in monitoring Tomcat using the JMX integration, define an event_type called “TomcatSample” and query the data in Insights using that event type.

For more on how to find and use your integrations data in Insights, see Use integration data in Insights.

Metric data

The metrics generated by the integration include metadata associated with the MBean they are collecting from. This metadata should be used in NRQL queries to filter and facet the data so that the query returns only the data for the desired beans. It can also be used to uniquely identify the metrics, since the metric name is not necessarily unique between all beans.

Each event contains the following metadata:

Name Description

displayName

The JMX domain name for these metrics.

entityName

The JMX domain name for these metrics with the entity type “domain:” prepended.

host

The JMX host the metrics are being collected from.

query

The query used to collect these metrics.

bean

The bean whose attributes these metrics were collected from.

key:<mbean_key>

For each key in the bean name, an attribute is added to the metric set called “key:<mbean_key> with the value of the bean’s key.

Below is an example NRQL query taking advantage of metadata monitor all the collected JVM garbage collectors:

SELECT latest(CollectionTime) 
FROM JVMSample 
FACET `key:name` 
WHERE `key:type` = 'GarbageCollector'

The JMX integration collects the following metric data attributes:

Name Description

HeapMemoryUsage.Used

The total Java heap memory used.

HeapMemoryUsage.Committed

The total Java heap memory committed to be used.

HeapMemoryUsage.Init

The initial Java heap memory allocated.

HeapMemoryUsage.Max

The maximum Java heap memory available.

NonHeapMemoryUsage.Used

The total Java non-heap memory used.

NonHeapMemoryUsage.Committed

The total Java non-heap memory committed to be used.

NonHeapMemoryUsage.Init

The initial Java non-heap memory allocated.

NonHeapMemoryUsage.Max

The maximum Java non-heap memory available.

ThreadCount

The number of live threads.

CollectionCount

The total number of garbage collections that have occurred.

CollectionTime

The approximate accumulated garbage collection time elapsed.

Inventory data

The JMX integration captures the configuration parameters of the JMX integration. The data is available on the Infrastructure Inventory page, under the config/jmx source. For more about inventory data, see Understand integration data.

Troubleshooting

Troubleshooting tips:

Search logs for errors

If you are having trouble with the integration, first enable and search the logs for errors.

Metrics limit exceeded

If you suspect there is a domain sending more than 200 metrics, you can check the log file for this message:

"Domain x has n metrics, the current limit is 200. This domain will not be reported."

If you see this error message, lower the number of metrics being sent for the reported domain.

Missing metrics

If you have missing metrics, ensure that the MBean query is valid by attempting to run it with the nrjmx tool, or use your preferred tool for ensuring the query is valid in the JMXConsole.

Dashboard not appearing in Infrastructure

Confirm that the jvm-metrics.yml file has been updated, and that the path to the file is enumerated in the jmx-config.yml file.

For more help

Recommendations for learning more: