Infrastructure agent self-instrumentation

From agent version 1.24.0 you can enable agent self-instrumentation and monitor the agent performance using New Relic APM. This will allow you to have a better vision of the agent's behaviour, queues utilization and integrations payloads among others making it easier to configure and troubleshoot it.

Important

Infrastructure agent self-instrumentation is an experimental feature. The instrumented telemetry might change (metrics, transactions, custom events). We recommend to enable it only for complex troubleshooting scenarios. Standard pricing for data ingest applies.

Self instrumentation is disabled by default. To enable it use the configuration option self_instrumentation:

self_instrumentation: "newrelic"

Once self_instrumentation is enabled, the agent will report metrics about itself and a new application New Relic infrastructure agent will appear in the New Relic APM UI.

Infrastructure agent APM application in entity explorer

Accessing the New Relic infrastructure agent application you'll see data for:

Go Runtime (cpu, memory...)
Transactions (samplers, integrations, https post...)

To see custom metrics about the agent's internal Event Queue utilization and Integrations payloads you can import the following dashboard.

Tip

Replace ACCOUNT_ID placeholder with your account number in the json payload below. More information about importing dashboards

{
  "name": "New Relic infrastructure agent instrumentation",
  "description": "Overview of New Relic infrastructure agent",
  "permissions": "PUBLIC_READ_WRITE",
  "pages": [
    {
      "name": "Overview",
      "description": null,
      "widgets": [
        {
          "title": "Goroutines",
          "layout": {
            "column": 1,
            "row": 1,
            "width": 4,
            "height": 3
          },
          "linkedEntityGuids": null,
          "visualization": {
            "id": "viz.line"
          },
          "rawConfiguration": {
            "facet": {
              "showOtherSeries": false
            },
            "legend": {
              "enabled": true
            },
            "nrqlQueries": [
              {
                "accountId": ACCOUNT_ID,
                "query": "SELECT average(newrelic.timeslice.value) AS `Go/Runtime/Goroutines` FROM Metric WHERE metricTimesliceName = 'Go/Runtime/Goroutines' AND appName =  'New Relic Infrastructure Agent' TIMESERIES"
              }
            ],
            "yAxisLeft": {
              "zero": true
            }
          }
        },
        {
          "title": "Agent Used Memory",
          "layout": {
            "column": 5,
            "row": 1,
            "width": 4,
            "height": 3
          },
          "linkedEntityGuids": null,
          "visualization": {
            "id": "viz.line"
          },
          "rawConfiguration": {
            "facet": {
              "showOtherSeries": false
            },
            "legend": {
              "enabled": true
            },
            "nrqlQueries": [
              {
                "accountId": ACCOUNT_ID,
                "query": "FROM Metric SELECT average(apm.service.memory.physical) WHERE appName = 'New Relic Infrastructure agent'  TIMESERIES"
              }
            ],
            "yAxisLeft": {
              "zero": true
            }
          }
        },
        {
          "title": "Integrations Payload Size",
          "layout": {
            "column": 9,
            "row": 1,
            "width": 4,
            "height": 3
          },
          "linkedEntityGuids": null,
          "visualization": {
            "id": "viz.stacked-bar"
          },
          "rawConfiguration": {
            "facet": {
              "showOtherSeries": false
            },
            "legend": {
              "enabled": true
            },
            "nrqlQueries": [
              {
                "accountId": ACCOUNT_ID,
                "query": "FROM Transaction SELECT average(payload_size) facet name,integration_hostname,integration_port LIMIT MAX TIMESERIES"
              }
            ]
          }
        },
        {
          "title": "Queue Event utilization",
          "layout": {
            "column": 1,
            "row": 4,
            "width": 4,
            "height": 3
          },
          "linkedEntityGuids": null,
          "visualization": {
            "id": "viz.line"
          },
          "rawConfiguration": {
            "legend": {
              "enabled": true
            },
            "nrqlQueries": [
              {
                "accountId": ACCOUNT_ID,
                "query": "FROM Metric SELECT percentile(agent.eventQueueUtilization,99,95,50) TIMESERIES "
              }
            ],
            "yAxisLeft": {
              "zero": true
            }
          }
        },
        {
          "title": "Queue Capacity",
          "layout": {
            "column": 5,
            "row": 4,
            "width": 4,
            "height": 3
          },
          "linkedEntityGuids": null,
          "visualization": {
            "id": "viz.line"
          },
          "rawConfiguration": {
            "legend": {
              "enabled": true
            },
            "nrqlQueries": [
              {
                "accountId": ACCOUNT_ID,
                "query": "FROM Metric SELECT max(agent.eventQueueCapacity) TIMESERIES"
              }
            ],
            "yAxisLeft": {
              "zero": true
            }
          }
        },
        {
          "title": "postEventsNum",
          "layout": {
            "column": 9,
            "row": 4,
            "width": 4,
            "height": 3
          },
          "linkedEntityGuids": null,
          "visualization": {
            "id": "viz.line"
          },
          "rawConfiguration": {
            "legend": {
              "enabled": true
            },
            "nrqlQueries": [
              {
                "accountId": ACCOUNT_ID,
                "query": "FROM Metric SELECT max(agent.postEventsNum) TIMESERIES"
              }
            ],
            "yAxisLeft": {
              "zero": true
            }
          }
        },
        {
          "title": "Transactions",
          "layout": {
            "column": 1,
            "row": 7,
            "width": 4,
            "height": 3
          },
          "linkedEntityGuids": null,
          "visualization": {
            "id": "viz.line"
          },
          "rawConfiguration": {
            "facet": {
              "showOtherSeries": false
            },
            "legend": {
              "enabled": true
            },
            "nrqlQueries": [
              {
                "accountId": ACCOUNT_ID,
                "query": "FROM Transaction SELECT average(duration) WHERE appName = 'New Relic Infrastructure agent' FACET name TIMESERIES "
              }
            ],
            "yAxisLeft": {
              "zero": true
            }
          }
        },
        {
          "title": "Transaction Errors",
          "layout": {
            "column": 5,
            "row": 7,
            "width": 4,
            "height": 3
          },
          "linkedEntityGuids": null,
          "visualization": {
            "id": "viz.line"
          },
          "rawConfiguration": {
            "legend": {
              "enabled": true
            },
            "nrqlQueries": [
              {
                "accountId": ACCOUNT_ID,
                "query": "FROM TransactionError SELECT count(*) TIMESERIES"
              }
            ],
            "yAxisLeft": {
              "zero": true
            }
          }
        }
      ]
    }
  ]
}