JMX monitoring integration

Our JMX integration allows users to monitor any application that exposes metrics with JMX. The integration includes a default collection file. It collects key metrics from the JVM. The collected metrics vary by app server and include measurements of thread pools, HTTP sessions, and transactions.

Read on to install the integration and to see what data we collect.

Tip

If you're not running JMX in Kubernetes or ECS environments, we recommend our guided install. Our guided install uses our infrastructure agent and our CLI to set up the JMX integration. It also finds other applications and log sources in your environment. Then, it recommends which ones you should instrument. The guided install works with most setups. But if it doesn't suit your needs, there are other install options below.

Ready to get started? Click the relevant button, depending on which data center region you use. When you're done with the install, return to this document to review the configuration options.

Guided install, US region

Guided install, EU region

For a more permanent and scalable solution, we recommend the standard manual install of the agent. Keep reading for how to do that.

Important

This integration doesn't automatically update. For best results, regularly update the integration package and the infrastructure agent.

Configuration options

For the standard on-host installation, this integration comes with a YAML config file, jmx-config.yml. This configuration is where you can place required login credentials and configure how data is collected. Which options you change depends on your setup and preferences. It comes with a sample config file jmx-config.yml.sample that you can copy and edit.

Specific settings related to JMX are defined using the env section of the configuration file. These settings control the connection to your JMX instance as well as other security settings and features. The list of valid settings is described in the next section of this document.

The JMX integration collects both metrics and inventory information. This table shows what each configuration option applies to.

Setting	Description	Default	Applies To
`JMX_HOST`	The host JMX is running on.	`localhost`	M/I
`JMX_PORT`	The port JMX is running on.	`9999`	M/I
`JMX_URI_PATH`	The path portion of the JMX Service URI. This is useful for nonstandard service uris.	N/A	M/I
`JMX_USER`	The username for the JMX connection.	N/A	M/I
`JMX_PASS`	The password for the JMX connection.	N/A	M/I
`JMX_REMOTE`	(JBoss specific) Whether or not to use the JMX remote URL connection format. Connection defaults to JBoss Domain-mode if `true`.	`false`	M/I
`JMX_REMOTE_JBOSS_STANDLONE`	(JBoss specific) Whether or not to use the JBoss standalone connection format. Only relevant if `JMX_REMOTE` is set.	`false`	M/I
`CONNECTION_URL`	Full JMX endpoint URL. This replaces all connection arguments (above) by providing all parameters on one line. Example: `"service:jmx:rmi:///jndi/rmi://localhost:7199/jmxrmi"`	N/A	M/I
`COLLECTION_FILES`	A comma-separated list of full file paths to the metric collection definition files. For on-host install, the default JVM metrics collection file is at `/etc/newrelic-infra/integrations.d/jvm-metrics.yml`.	N/A	M/I
`COLLECTION_CONFIG`	Configuration of the metrics collection as a JSON. Jvm Example: COLLECTION_CONFIG='{"collect":[{"domain":"java.lang","event_type":"JVMSample","beans":[{"query":"type=GarbageCollector,name=*","attributes":["CollectionCount","CollectionTime"]},{"query":"type=Memory","attributes":["HeapMemoryUsage.Committed","HeapMemoryUsage.Init","HeapMemoryUsage.Max","HeapMemoryUsage.Used","NonHeapMemoryUsage.Committed","NonHeapMemoryUsage.Init","NonHeapMemoryUsage.Max","NonHeapMemoryUsage.Used"]},{"query":"type=Threading","attributes":["ThreadCount","TotalStartedThreadCount"]},{"query":"type=ClassLoading","attributes":["LoadedClassCount"]},{"query":"type=Compilation","attributes":["TotalCompilationTime"]}]}]}' Tomcat Example: `COLLECTION_CONFIG={"collect":[{"domain":"Catalina","event_type":"TomcatSample","beans":[{"query":"type=UtilityExecutor","attributes":["completedTaskCount"]}]}]}` `COLLECTION_CONFIG` is useful to configure `nri-jmx` in Kubernetes using annotations.	N/A	M/I
`KEY_STORE`	The filepath of the keystore containing the JMX client's SSL certificate.	N/A	M/I
`KEY_STORE_PASSWORD`	The password for the SSL key store.	N/A	M/I
`LOCAL_ENTITY`	Collect all metrics on the local entity. Only use when monitoring localhost.	`false`	M/I
`TIMEOUT`	The timeout for individual JMX queries, in milliseconds.	`10000`	M/I
`TRUST_STORE`	The filepath of the keystore containing the JMX server's SSL certificate.	N/A	M/I
`TRUST_STORE_PASSWORD`	The password for the trust store.	N/A	M/I
`METRIC_LIMIT`	Number of metrics that can be collected per entity. If this limit is exceeded the entity will not be reported. A limit of `0` implies no limit.	`200`	M/I
`METRICS*`	Set to `true` to enable metrics only collection.	`false`
`INVENTORY`	Set to `true` to enable inventory only collection.	`false`

You can define the values for these settings in several ways.

Adding the value directly in the config file. This is the most common way.
Replacing the values from environment variables using the {{}} notation. This requires infrastructure agent v1.14.0 or higher. Read more here.
Using secrets management. Use this to protect sensible information (such as passwords) from being exposed in plain text on the configuration file. For more information, see secrets management.

You can customize your labels, too. Add key or value pair attributes to your metrics, which you can then use to query, filter or group your metrics on.

Our default sample config file has examples of labels. These aren't mandatory, and you can remove, modify, or add new ones of your choice.

labels:
  env: production
  role: jmx

Metrics collection files

The metrics collection definition files are structured YAML files which tell the integration what metrics to collect.

This is the default JVM metrics collection file: /etc/newrelic-infra/integrations.d/jvm-metrics.yml

Tip

You can write different collection files to ease organization and maintenance. See configuration file for an example.

The integration collects and organizes metrics according to domains. You can find all metrics defined per domain sent to New Relic in a corresponding event type. This event type is either auto-generated or can be set by the user. Each file contains a single collect: block which contains an array of domains. For each domain, these keys are defined:

domain: The JMX domain; for example, java.lang. You can use wildcards to match multiple domains; for example, java.*. If you use a wildcard, event_type is required, and must be unique. This field is required.
event_type: The event type name for a collection from this domain. If the domain is wildcarded, this is required, and must be unique. If the domain is not wildcarded and this is undefined by the user, this will be auto generated. For example, the domain java.lang will have event type JavaLangSample. For more information, see Naming tips.
beans: An array of beans to collect in this domain.
Important
There is a limit of 200 metrics per instance in the configuration file. If you exceed the limit for a particular instance, New Relic won't receive it. If you're not seeing your data in New Relic, review the troubleshooting procedures to identify if you've exceeded the limit.

Each domain has a variety of beans that you can collect. For each bean, these keys are defined:

query: The bean name to collect; for example,type=GarbageCollector,name=YoungGen. You can use wildcards; for example, type=GarbageCollector,name=*. This field is required.
exclude_regex: An optional list of regex patterns that match beans to exclude from collection; for example, type=GarbageCollector,name=.*.
attributes: A list of attributes to collect. If unspecified, collects all attributes.
Important
The HashMap and ArrayList data types are not supported.

Each bean can contain attributes and may have an optional list of beans that it can exclude from collection. For each attribute, these keys are defined:

Important

For map attributes, you must define either an attr or an attr_regex key.

attr: An exact match of the attribute name. You can collect composite attributes by appending the composite member name to the attribute name with a dot; for example, HeapMemoryUsage.Max.
attr_regex: A regex pattern that matches the attributes to be collected.
metric_type: The New Relic metric type to collect this attribute. These are the options:
- gauge: We will collect data as an instantaneous numeric measurement.
- rate: We will collect data showing the change in that metric per second.
- delta: We will collect data on the change in that metric since the last measurement.
- attribute: We will collect data as a string literal.
If left unspecified, the JMX integration will attempt to infer the metric type based on the value returned. For example, if the metric is a number, it will collect it as gauge. If the metric is a string, it will collect it as attribute.
If metrics are collected with an incorrect metric type, you can manually specify the correct metric type in the collection file.
metric_name: The name under which the metric will appear in New Relic. If unspecified, it will default to the attribute name.
For more information about JMX queries, see the Oracle ObjectName documentation.

Collection configuration using Kubernetes annotations

You can use Kubernetes annotations to provide collection configuration. To achieve this, you need to deploy a configMap into the Kubernetes cluster that will create the configuration file for the nri-jmx application.

In this configuration file you need to specify the command for container auto-discovery, which will allow you to use placeholders in the integration configuration, including Kubernetes annotations.

Example of a configMap to monitor JVM in a Tomcat application:

nri-bundle:
  newrelic-infrastructure:
    integrations_config:
      - name: jmx-config.yml
        data:
          discovery:
            command:
              # Use the following optional arguments:
              # --namespaces: Comma separated list of namespaces to discover pods on
              # --tls: Use secure (TLS) connection
              # --port: Port used to connect to the kubelet. Default is 10255
              exec: /var/db/newrelic-infra/nri-discovery-kubernetes
              match:
                label.app: java
          integrations:
            - name: nri-jmx
              env:
                # Using the discovered IP as the host address
                JMX_HOST: ${discovery.ip}
                JMX_PORT: ${discovery.port}
                COLLECTION_CONFIG: ${discovery.annotation.newrelic.config}

Learn more about how to monitor on-host integrations here

apiVersion: v1
kind: ConfigMap
metadata:
  name: nri-integration-cfg
data:
  jmx-config.yml: |
    ---
    # Run auto discovery to find pods with label "app=java"
    discovery:
      command:
        # Use the following optional arguments:
        # --namespaces: Comma separated list of namespaces to discover pods on
        # --tls: Use secure (TLS) connection
        # --port: Port used to connect to the kubelet. Default is 10255
        exec: /var/db/newrelic-infra/nri-discovery-kubernetes
        match:
          label.app: java
    integrations:
      - name: nri-jmx
        env:
          # Using the discovered IP as the host address
          JMX_HOST: ${discovery.ip}
          JMX_PORT: ${discovery.port}
          COLLECTION_CONFIG: ${discovery.annotation.newrelic.config}

Next, use annotations to define collection configuration. For example, here's a Tomcat deployment with annotations:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-deployment
  labels:
    app: java
spec:
  replicas: 1
  selector:
    matchLabels:
      app: java
  template:
    metadata:
      annotations:
        newrelic.config: >-
          {
            "collect": [
              {
                "domain": "java.lang",
                "event_type": "JVMSample",
                "beans": [
                  {
                    "query": "type=GarbageCollector,name=*",
                    "attributes": [
                      "CollectionCount",
                      "CollectionTime"
                    ]
                  },
                  {
                    "query": "type=Memory",
                    "attributes": [
                      "HeapMemoryUsage.Committed",
                      "HeapMemoryUsage.Init",
                      "HeapMemoryUsage.Max",
                      "HeapMemoryUsage.Used",
                      "NonHeapMemoryUsage.Committed",
                      "NonHeapMemoryUsage.Init",
                      "NonHeapMemoryUsage.Max",
                      "NonHeapMemoryUsage.Used"
                    ]
                  },
                  {
                    "query": "type=Threading",
                    "attributes": [
                      "ThreadCount",
                      "TotalStartedThreadCount"
                    ]
                  },
                  {
                    "query": "type=ClassLoading",
                    "attributes": [
                      "LoadedClassCount"
                    ]
                  },
                  {
                    "query": "type=Compilation",
                    "attributes": [
                      "TotalCompilationTime"
                    ]
                  }
                ]
              }
            ]
          }
      labels:
        app: java
    spec:
      containers:
        - name: tomcat
          image: tomcat:10.0.12
          ports:
            - containerPort: 9999
          env:
            - name: CATALINA_OPTS
              value: '-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false'

Custom connectors

JMX allows the use of custom connectors to communicate with the application. In order to use a custom connector, you have to include the custom connectors in the nrjmx classpath.

By default, the sub-folder connectors is in the classpath. If this folder doen't exist, create it under the folder where nrjmx is installed.

For example, to add support for JBoss, create a folder named connectors under the default (Linux) library path /usr/lib/nrjmx/ (/usr/lib/nrjmx/connectors/) and copy the custom connector jar ($JBOSS_HOME/bin/client/jboss-cli-client.jar) into it. You can now execute JMX queries against JBoss.

Example configurations

Example file configurations for an on-host install:

integrations:
  - name: nri-jmx
    env:
      COLLECTION_FILES: "/etc/newrelic-infra/integrations.d/jvm-metrics.yml,/etc/newrelic-infra/integrations.d/tomcat-metrics.yml"
      JMX_HOST: jmx-host.localnet
      JMX_PASS: admin
      JMX_PORT: 9999
      JMX_USER: admin
    interval: 15s
    labels:
      env: production

collect:
  # The event type for this domain will be JavaLangSample
  - domain: java.lang
    beans:
      # Collect all beans of type Threading
      - query: type=Threading
        # Attributes can be either a string or a map
        attribute:
          # When unspecified, the metric_type is inferred
          # and the metric name is just the attribute name
          - ThreadCount
            # If using a map attribute, a custom metric name can be set
          - attr: TotalStartedThreadCount
            metric_name: ThreadsStarted
            # Attributes can be collected with regex matches and
            # the metric type can be overridden if the integration
            # can not correctly infer the type
          - attr_regex: "ThreadCpu.*Enabled"
            metric_type: attribute
      - query: type=Memory
        attributes:
          # Composite attributes can be collected with this syntax
          - HeapMemoryUsage.Max
          - NonHeapMemoryUsage.Max
        # Queries can be wildcarded where
      - query: type=GarbageCollector,name=*
        # If a specific bean is unwanted, it can be excluded
        # with a regex match pattern. Useful if using a wildcard query
        exclude_regex:
          # This will match any bean where the name is YoungGen
          - name=YoungGen
        attributes:
          - attr: LastGcInfo.GcThreadCount
            metric_type: gauge
            metric_name: GCThreadCount
    # Domains can be wildcarded
  - domain: java.util.*
    # If the domain is wildcarded, a custom event must be defined
    event_type: JavaUtilSample
    beans:
      # If no attributes are defined, all are collected by default
      - query: type=Logging

For more about the general structure of on-host integration configuration, see Configuration.

Naming your data

We send and store metrics in the form of samples. This is a list of key-value pairs that include metric data and metadata. Each sample is stored as an event in our database.

You are responsible for creating and naming the JMX data reported to New Relic. For this reason, New Relic strongly recommends following these conventions when naming your event types. To ensure you have a consistent naming scheme:

Use camel case.
Use a name that clearly identifies what data it contains.

Example: MyorgApplicationSample

Recommendation: Use the same naming scheme for similar metrics across different applications.

Metric data

The metrics show metadata for the MBean they are collecting from. You can use this metadata in NRQL queries to filter and facet the data so that the query returns only the data for the desired beans. You can also use it to identify the metrics, as the metric name may not be unique among all beans.

Each event contains the following metadata:

Name	Description
`displayName`	The JMX domain name for these metrics.
`entityName`	The JMX domain name for these metrics with the entity type `domain:` prepended.
`host`	The JMX host the metrics are being collected from.
`query`	The query used to collect these metrics.
`bean`	The bean whose attributes these metrics were collected from.
`key:<mbean_key>`	For each key in the bean name, an attribute is added to the metric set called `key:<mbean_key>` with the value of the bean's key.

Example NRQL query

Here's an example of NRQL query taking advantage of metadata monitor all the collected JVM garbage collectors:

SELECT latest(CollectionTime)
FROM JVMSample
FACET `key:name`
WHERE `key:type` = 'GarbageCollector'

Metrics data attributes

The JMX integration collects these metric data attributes:

Name	Description
`HeapMemoryUsage.Used`	The total Java heap memory used.
`HeapMemoryUsage.Committed`	The total Java heap memory committed to be used.
`HeapMemoryUsage.Init`	The initial Java heap memory allocated.
`HeapMemoryUsage.Max`	The maximum Java heap memory available.
`NonHeapMemoryUsage.Used`	The total Java non-heap memory used.
`NonHeapMemoryUsage.Committed`	The total Java non-heap memory committed to be used.
`NonHeapMemoryUsage.Init`	The initial Java non-heap memory allocated.
`NonHeapMemoryUsage.Max`	The maximum Java non-heap memory available.
`ThreadCount`	The number of live threads.
`CollectionCount`	The total number of garbage collections that have occurred.
`CollectionTime`	The approximate accumulated garbage collection time elapsed.

Inventory data

The JMX integration captures the configuration parameters of the JMX integration. The data is available on the Inventory page, under the config/jmx source. For more about inventory data, see Understand integration data.

Troubleshooting

Troubleshooting tips:

If you suspect there is a domain sending more than 200 metrics, check the log file for this message:

"Domain x has n metrics, the current limit is 200. This domain will not be reported."

If you see this error message, lower the number of metrics being sent for the reported domain.

By default the integration looks for the java binary into the PATH that the Infrastructure Agent has visibility on. In case java is not present in the PATH the JAVA_HOME environment variable must be set into the nri-jmx config file.

integrations:
- name: nri-jmx
  env:
    COLLECTION_FILES: "/etc/newrelic-infra/integrations.d/jvm-metrics.yml"
    JMX_HOST: jmx-host.localnet
    JMX_PASS: admin
    JMX_PORT: 9999
    JMX_USER: admin
    JAVA_HOME: "path/to/java/home"
  interval: 15s
  labels:
    env: production

Check the source code

This integration is open source software. That means you can browse its source code and send improvements, or create your own fork and build it.

JMX monitoring integration

Tip

Important

Configuration options

JMX Instance settings

Labels and custom attributes

Metrics collection files

Tip

Domains

Important

Beans

Important

Attributes

Important

Collection configuration using Kubernetes annotations

Example of configMap using helm charts (recommended)

Example of configMap using plain manifest

Custom connectors

Example configurations

Example host connection file

Example metrics collection file

Naming your data

Metric data

Example NRQL query

Metrics data attributes

Inventory data

Troubleshooting

Search logs for errors

Metrics limit exceeded

Missing metrics

Dashboard not appearing in infrastructure UI

Troubleshooting via JMXTerm

Java binary not found on the PATH

Check the source code

JMX monitoring integration

Tip

Important

Configuration options .css-21sua1{background:none;border:none;width:0;padding:0;}

Labels and custom attributes

Metrics collection files

Tip

Domains

Beans

Attributes

Collection configuration using Kubernetes annotations

Example of configMap using helm charts (recommended)

Example of configMap using plain manifest

Custom connectors

Example configurations

Example host connection file

Example metrics collection file

Naming your data

Metric data

Example NRQL query

Metrics data attributes

Inventory data

Troubleshooting

Search logs for errors

Metrics limit exceeded

Missing metrics

Dashboard not appearing in infrastructure UI

Troubleshooting via JMXTerm

Java binary not found on the PATH

Check the source code

Configuration options