StatsD monitoring integration (version 2)

The second version of our StatsD integration was released on May 25 2020. It's an improvement on our first StatsD integration. Improvements include: simpler configuration, and a simpler method for adding tags (key-value pairs).

Features

Our StatsD integration (version 2) lets you easily get StatsD-format data into New Relic. You can also add any arbitrary tags (key-value pairs) to your data. Once your metrics are in New Relic, you can query your data and create custom charts and dashboards.

Requirements

This integration uses our Metric API and our Event API to ingest data. To use these APIs, you'll need an Insert API key.

The integration adheres to the Metric API requirements and data limits. The default rate limit is 100,000 data points per minute (DPM). If you think you're missing metrics or sending more than 100K DPM, see Request data changes. To see if you're account is hitting the rate limit, run the following NRQL query:

SELECT count(*) FROM NrIntegrationError 
  WHERE newRelicFeature ='Metrics' 
  FACET category, message
  LIMIT 100 since 1 day ago

Install

This section will explain how to do a standard install. If you want to run StatsD in Kubernetes, see Kubernetes install.

To install the StatsD integration, run the following command and include your New Relic account ID and New Relic Insert API key. This generates a TOML configuration file used by gostatsd.

docker run \
  -d --restart unless-stopped \
  --name newrelic-statsd \
  -h $(hostname) \
  -e NR_ACCOUNT_ID=YOUR_ACCOUNT_ID \
  -e NR_API_KEY=YOUR_INSERT_API_KEY \
  -p 8125:8125/udp \
  newrelic/nri-statsd:2.0.0

If your account is in the EU data center region, add this to the above command:

-e NR_EU_REGION=true \

After installing, you can:

Install for Kubernetes

Here are examples of Kubernetes manifests for deployment and service objects:

Kubernetes manifest examples

Below are examples of Kubernetes manifests to deploy StatsD in a Kubernetes environment and create a StatsD service named newrelic-statsd. You need to insert your account ID and your license key.

deployment.yml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: newrelic-statsd
  namespace: tooling
  labels:
    app: newrelic-statsd
spec:
  selector:
    matchLabels:
      app: newrelic-statsd
  replicas: 2
  revisionHistoryLimit: 2
  template:
    metadata:
      labels:
        app: newrelic-statsd
    spec:
      containers:
      - name: newrelic-statsd
        image: newrelic/nri-statsd:2.0.0
        env:
        - name: NR_ACCOUNT_ID
          value: "NEW_RELIC_ACCOUNT_ID"
        - name: NR_API_KEY
          value: "NEW_RELIC_LICENSE_KEY"

service.yml:

apiVersion: v1
kind: Service
metadata:
  name: newrelic-statsd
  namespace: tooling
  labels:
    app: newrelic-statsd
spec:
  type: ClusterIP
  ports:
  - name: newrelic-statsd
    port: 80
    targetPort: 8125
    protocol: UDP
  selector:
    app: newrelic-statsd

For configuration details, see Kubernetes configuration.

Configure

In the install procedure, you run nri-statsd with environment variables, and this generates a TOML configuration file. Additionally, you can set these configuration options:

Configuration options Description
expiry-interval
string

If a metric is not updated for this amount of time, we stop reporting that metric. Default is 5m.

If you want to send the metrics only if the value was updated between the flush intervals, configure this to 1ms. To never expire metrics, set it to 0.

percent-threshold
list of integers
Specifies the percentiles used for metrics aggregation. Default: 90.
metrics-addr
string
Indicates address on which to listen for metrics. Default: :8125.

To ensure FedRAMP compliance when using the StatsD integration you must define the following endpoints in the custom configuration:

address = 'https://gov-insights-collector.newrelic.com/v1/accounts/$NR_ACCOUNT_ID/events'

address-metrics = 'https://gov-infra-api.newrelic.com/metric/v1'

Here are some examples of customizing configuration by overwriting the default configuration:

Example of custom configuration
# Specify after how long do we expire metrics, default:5m
expiry-interval = '1ms'

# percent-threshold specify a list of percentiles for metrics aggregation, default:90
percent-threshold = [90, 99]

backends='newrelic'
[newrelic]
# flush types supported: metrics,  insights, infra
flush-type = 'metrics'
transport = 'default'
address = 'https://insights-collector.newrelic.com/v1/accounts/$NR_ACCOUNT_ID/events'
address-metrics = 'https://metric-api.newrelic.com/metric/v1'
api-key = 'NEW_RELIC_API_KEY'
Disable timer sub-metrics:

By default, nri_statsd calculates the following for timer metrics: standard deviation, mean, median, sum, lower, and upper bounds for the flush interval. If you want to disable those metrics you can do it by adding a disabled-sub-metrics configuration section and set true for the ones you want disabled. Here's an example:

# disabled-sub-metrics configuration section allows disabling timer sub-metrics
[disabled-sub-metrics]
# Regular metrics
count=false
count-per-second=false
mean=false
median=false
lower=false
upper=false
stddev=false
sum=false
sum-squares=false

# Percentile metrics
count-pct=false
mean-pct=false
sum-pct=false
sum-squares-pct=false
lower-pct=false
upper-pct=false

To overwrite the default nri-statsd configuration while running in a container, you can mount a configuration file inside the container.

You can adopt the following template as needed for your situation.

Example:

backends='newrelic'
flush-interval='10s'

[newrelic]
# flush types supported: metrics,  insights, infra
flush-type = 'metrics'
transport = 'default'
address-metrics = 'https://metric-api.newrelic.com/metric/v1'
api-key = 'NEW_RELIC_API_KEY'

To run the container with the file mounted in the appropriate path:

docker run \
  ...
  -v ${PWD}/nri-statsd.toml:/etc/opt/newrelic/nri-statsd.toml \
  ...
  newrelic/nri-statsd:2.0.0

The best approach to configure nri-statsd running in Kubernetes is to use a configMap and mount the configMap into the container. (This is a similar process to mounting the configuration file in Docker.)

Example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nri-statsd-config
  namespace: default
data:
  nri-statsd.toml: |
    backends='newrelic'
    flush-interval='10s'

    [newrelic]
    # flush types supported: metrics,  insights, infra
    flush-type = 'metrics'
    transport = 'default'
    address = 'https://metric-api.newrelic.com/metric/v1'
    api-key = '$NEW_RELIC_API_KEY'

To use the configMap, declare a volume on your deployment spec template and then declare a volumeMount on your container spec.

Example:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
    containers:
      ....
      volumeMounts:
        - mountPath: /etc/opt/newrelic/
          name: nri-statsd-config
    volumes:
      - name: nri-statsd-config
        configMap:
          name: nri-statsd-config

Metric format

The integration receives metrics using the StatsD protocol. Optionally, the sample rate can be configured and tags can be added.

Here's the metric data format we use:

<metric name>:<value>|<type>|@<sample rate>|#<tags>

Here are explanations of these fields:

Field name Description
<metric name>
string
Required. Name of the metric.
<value>
string
Required. The metric type:
  • c = counter
  • g = gauge
  • ms = timer
@<sample rate>
float
Optional for simple counters or timer counters. When many metrics must be sent, you can use sampling to reduce network traffic. The downside is a reduction in the resolution of the data.

An example of how this would work for sample rates below 1: If you set this to 0.1, the counter would send a measurement one out of every 10 times.

#<tags>
string
Optional. Tags attached to your metrics are converted into attributes (key-value pairs). For more on tagging options, see Tags.

Metric types

Here are the types of metrics and how to format them:

Counter

A counter measures the number of occurrences of an event. Examples include cache hits per reporting interval and the number of threads created per reporting interval.

A counter can be incremented or decremented during the same flush interval by adding a sign to the value. In the following example, the counter value will be 2:

 counter:4|c
 counter:-2|c 

At each flush, the current count is sent and reset to 0. If the count is not updated, at the next flush it will send the value 0. You can opt to disable this behavior by setting expiry-interval to 1ms.

Here’s an example of a counter that is being sampled 1 out of 10 times:

counter:4|c@0.1
Gauge

A gauge represents a value that can increase or decrease with time. Examples of gauges include temperature, CPU usage, and memory. Here's an example:

temperature:40|g

If the gauge is not updated, at the next flush it will send the previous value. You can opt to disable this behavior by setting expiry-interval to 1ms.

Timer

The timer metric type measures timing data.

By default, nri_statsd calculates the following for timer metrics: standard deviation, mean, median, sum, lower, and upper bounds for the flush interval. These are sent as sub-metrics in the following format:

<metric_base_name>.std_dev 
<metric_base_name>.median
<metric_base_name>.summary
<metric_base_name>.sum_squares
<metric_base_name>.mean
<metric_base_name>.per_second

The configured percentiles will generate the following metrics. The percentile threshold value will be attached as a tag.

<metric_base_name>.sum_squares.percentiles
<metric_base_name>.sum.percentiles
<metric_base_name>.count.percentiles
<metric_base_name>.upper.percentiles
<metric_base_name>.mean.percentiles

The percentile threshold can be tweaked with the percent-threshold config option. These can be controlled through the disabled-sub-metrics configuration section.

Add tags (attributes)

You can add tags to your data, which we save as attributes (key-value pairs). There are two options for adding tags:

  • Add default tags that apply to all metrics: These apply to all metrics. They are fixed and don't change over time.
  • Add metric-level tags: These apply to specific metrics and allow the value to be changed between two submits.
Add default tags that apply to all metrics

Add tags to metrics and events by defining an environment variable in the startup command.

Here's an example that would create two tags:

-e TAGS="environment:production region:us"

Here's that environment variable used in the startup command:

docker run \
  -d --restart unless-stopped \
  --name newrelic-statsd \
  -h $(hostname) \
  -e NR_ACCOUNT_ID=YOUR_ACCOUNT_ID \
  -e NR_API_KEY=YOUR_INSERT_API_KEY \
  -e TAGS="environment:production region:us" \ 
  -p 8125:8125/udp \
  newrelic/nri-statsd:2.0.0
Add metric-level tags

When defining the metric format, you can add tags using this format:

<bucket name>:<value>|<type>|#<tags> 

In this example, <tags> is a comma-separated list of tags. Tags format is: simple or key:value.

Here's an example NRQL query that includes a custom tag:

SELECT count(*) FROM Metric WHERE environment = 'production'

Create alerts

You can alert on StatsD data using NRQL alert conditions.

Alert example

This procedure walks you through sending some sample data and then creating an alert condition using that data.

First, send this data to New Relic’s StatsD container:

echo "prod.test.num:32|g" | nc -v -w 1 -u localhost 8125

Next, create a NRQL alert condition using this query:

SELECT latest(prod.test.num) FROM Metric WHERE metricName = 'prod.test.num'

Here's an image showing creating this NRQL alert condition. Notice that the sample data sent in is represented by the blue dot on the upper right of the chart.

StatsD NRQL alert condition query

Now we can create the alert condition with these settings:

StatsD NRQL alert condition creation

Remember to set the Condition name.

If a metric with a value above 50 is sent, then an incident is created and notified. The incident is closed automatically after 24 hours. To test that the alert is working, run this command:

echo "prod.test.num:60|g" | nc -v -w 1 -u localhost 8125

Find and use data

To query your data, you'd use any New Relic query option. For example, you might run a NRQL query like:

SELECT count(*) FROM Metric WHERE metricName = 'myMetric' and environment = 'production'

For more on how to query the Metric data type, see Query metric data.

Check the source code

This integration is open source software. That means you can browse its source code and send improvements, or create your own fork and build it.

For more help

If you need more help, check out these support and learning resources: