Get started with collector observability

Billing

Your use of Collector Observability is billable during preview in accordance with your Order as applicable to the pricing model associated with your Account and as defined below.

The costs associated with this feature are determined by the following factors, as applicable to the pricing model associated with your Account:

Core Compute: The Summary page, measured in Core CCU, is billable during preview.

Data Ingest: Additional data from the internal telemetry, measured in GB Ingested is billable during preview.

If this feature becomes generally available, your use will be billable in accordance with your Order.

Step-by-step configuration

Download the configuration file

By default, the collector doesn't emit its internal telemetry, so you'll need to enable it first.

bash

$curl 'https://raw.githubusercontent.com/newrelic/nrdot-collector-releases/refs/heads/main/examples/internal-telemetry-config.yaml' \
>  --silent --output internal-telemetry-config.yaml

Set environment variables

INTERNAL_TELEMETRY_NEW_RELIC_LICENSE_KEY: Ingest license key for the account internal telemetry should be sent to. This key can be different than the key the collector uses to send regular data to New Relic, i.e. NEW_RELIC_LICENSE_KEY in the example below.
INTERNAL_TELEMETRY_SERVICE_NAME: Defaults to otel-collector, determines entity name in New Relic
INTERNAL_TELEMETRY_OTLP_ENDPOINT: Defaults to US https://otlp.nr-data.net; if you are in EU, set this to https://otlp.eu01.nr-data.net; if you are in JP, set this to https://otlp.jp.nr-data.net

Run collector with merged configuration

In addition to the normal configuration for components and pipelines (in the following example --config=/etc/nrdot-collector/config.yaml), add a second --config argument which will merge both configurations:

bash

$docker run \
>  -e INTERNAL_TELEMETRY_NEW_RELIC_LICENSE_KEY='...' \
>  -e NEW_RELIC_LICENSE_KEY='...' \
>  -e INTERNAL_TELEMETRY_SERVICE_NAME='demo-collector' \
>  -v './internal-telemetry-config.yaml:/etc/nrdot-collector/config-internal.yaml' \
>  newrelic/nrdot-collector:2.5.0 --config=/etc/nrdot-collector/config.yaml \
>  --config='/etc/nrdot-collector/config-internal.yaml'

Important

The order of the --config arguments matters if you have preexisting configuration under the service::telemetry node. The collector uses a 'last one wins' strategy on a node level when merging configurations and certain parts of the config (e.g. lists, leaf nodes) cannot be merged, so they get overwritten by the last --config argument.

Add entity tag

The tag newrelic.service.type: otel_collector acts as an opt-in to the experience at the UI level. Choose one of the following options:

Option 1: Use the example configuration provided above which contains the configuration of Option 2.
Option 2: Add argument --config=yaml:service::telemetry::resource::newrelic.service.type: otel_collector to collector. This adds the attribute as a resource attribute and New Relic does the tagging for you on ingest. If you remove this option, it will take one day for the tag to expire.
Option 3: Add tag via APM UI (top of the page, next to entity name). You can remove this via the UI as well to toggle back.

Once tagged, see View your collector in the UI to explore your collector's data.

Optional: Customize the configuration

The default configuration exposes additional environment variables of the form INTERNAL_TELEMETRY_... to tweak common options such as detail levels and sampling. Refer to the configuration itself for more details.

We recommend using the default configuration to monitor your collector. However, you can modify the example configuration to meet your needs, such as reducing detail level, sampling or data collection rates. Refer to the official documentation for more details. Keep in mind that changes to the configuration may cause certain parts of the UI to not display data. Also refer to Limitations. The collector's internal telemetry configuration options change as the OTel community evolves. You have full control over your configuration, including whether to use the provided defaults.

Overhead considerations

Like all telemetry, the collector's internal telemetry increases your data ingest. The overhead depends on your workload and configuration. Here are the key factors to consider:

Metrics Creates constant overhead for the collector and all active components, regardless of throughput. Metrics are emitted at a constant interval (60 seconds by default). Any component can emit custom metrics, as indicated by the respective metadata.yaml, which may add to the overhead.

If you need to reduce the amount of metrics your collector emits, we recommend setting the metric level to normal, e.g. via the environment variable INTERNAL_TELEMETRY_METRICS_LEVEL in the example config, as only a subset of the detailed metrics are used by the UI and are typically metrics intended for fine-tuning performance or network issues, so you can re-enable them as needed.

Logs Has minimal impact during normal operation. Higher overhead can occur when error logs increase rapidly, but this is reduced because logs are sampled by default. The sampling algorithm allows a constant number of logs per defined interval and then samples at a static rate. Refer to service::telemetry::logs::sampling.

Traces Disabled by default due to:

Limited maturity of traces. Not all components are instrumented, see this github issue.
Unpredictable overhead that varies with workload and batching configuration from upstream agents and collectors.
No adaptive sampling algorithm available. This makes it impossible to provide a universal sampling recommendation that doesn't risk unexpected costs for certain use cases.

When traces become more mature, they will provide valuable insight into how much time the collector spends processing data at the component level. To experiment with traces as they develop, set INTERNAL_TELEMETRY_TRACE_LEVEL=info and start with a low sampling rate such as INTERNAL_TELEMETRY_TRACE_SAMPLE_RATIO=0.001 (0.1%) while monitoring your ingest volume and adjusting as needed.

Configuration examples

This section contains examples of how the above step-by-step guide translates to different use cases involving the collector.

Simple collector with internal telemetry enabled

Minimal example of a collector running with docker compose.

Populate infrastructure relationships (optional, experimental)

As mentioned on the Summary page, APM can show infrastructure metrics for the infrastructure the collector is running on but only if your infrastructure is instrumented and a relationship can be formed. The relationship is driven by additional attributes on the collector's internal telemetry. You can read more about this topic in our OTel documentation. The steps to set this up heavily depend on your exact infrastructure. For the common choice of Kubernetes we created an example that showcases how to achieve container-to-service relationships based on our OTel for Kubernetes solution.