--- title: Get started with collector observability source: https://docs.newrelic.com/docs/opentelemetry/collector-observability/collector-observability-setup --- > #### ⚠️ BILLING > > Your use of Collector Observability is billable during preview in accordance with your Order as applicable to the pricing model associated with your Account and as defined below. > > The costs associated with this feature are determined by the following factors, as applicable to the pricing model associated with your Account: > > **Core Compute**: The Summary page, measured in Core CCU, is billable during preview. > > **Data Ingest**: Additional data from the internal telemetry, measured in GB Ingested is billable during preview. > > If this feature becomes generally available, your use will be billable in accordance with your Order. ## Step-by-step configuration ### Download the configuration file [#download-config] By default, the collector doesn't emit its [internal telemetry](https://opentelemetry.io/docs/collector/internal-telemetry/), so you'll need to enable it first. ```bash curl 'https://raw.githubusercontent.com/newrelic/nrdot-collector-releases/refs/heads/main/examples/internal-telemetry-config.yaml' \ --silent --output internal-telemetry-config.yaml ``` ### Set environment variables [#set-env-vars] - `INTERNAL_TELEMETRY_NEW_RELIC_LICENSE_KEY`: Ingest license key for the account internal telemetry should be sent to. This key can be different than the key the collector uses to send regular data to New Relic, i.e. `NEW_RELIC_LICENSE_KEY` in the example below. - `INTERNAL_TELEMETRY_SERVICE_NAME`: Defaults to `otel-collector`, determines entity name in New Relic - `INTERNAL_TELEMETRY_OTLP_ENDPOINT`: Defaults to US `https://otlp.nr-data.net`; if you are in EU, set this to `https://otlp.eu01.nr-data.net`; if you are in JP, set this to `https://otlp.jp.nr-data.net` ### Run collector with merged configuration [#run-collector] In addition to the normal configuration for components and pipelines (in the following example `--config=/etc/nrdot-collector/config.yaml`), add a second `--config` argument which will merge both configurations: ```bash docker run \ -e INTERNAL_TELEMETRY_NEW_RELIC_LICENSE_KEY='...' \ -e NEW_RELIC_LICENSE_KEY='...' \ -e INTERNAL_TELEMETRY_SERVICE_NAME='demo-collector' \ -v './internal-telemetry-config.yaml:/etc/nrdot-collector/config-internal.yaml' \ newrelic/nrdot-collector:2.5.0 --config=/etc/nrdot-collector/config.yaml \ --config='/etc/nrdot-collector/config-internal.yaml' ``` > #### ⚠️ IMPORTANT > > The order of the `--config` arguments matters if you have preexisting configuration under the `service::telemetry` node. > The collector uses a 'last one wins' strategy on a node level when merging configurations and certain parts of the config (e.g. lists, leaf nodes) cannot be merged, so they get overwritten by the last `--config` argument. ### Add entity tag [#add-tag] The tag `newrelic.service.type: otel_collector` acts as an opt-in to the experience at the UI level. Choose one of the following options: - **Option 1**: Use the example configuration provided above which contains the configuration of Option 2. - **Option 2**: Add argument `--config=yaml:service::telemetry::resource::newrelic.service.type: otel_collector` to collector. This adds the attribute as a resource attribute and New Relic does the tagging for you on ingest. If you remove this option, it will take one day for the tag to expire. - **Option 3**: Add tag via APM UI (top of the page, next to entity name). You can remove this via the UI as well to toggle back. Once tagged, see [View your collector in the UI](https://docs.newrelic.com/docs/opentelemetry/collector-observability/collector-observability#view-ui) to explore your collector's data. ### Optional: Customize the configuration [#customize] The default configuration exposes additional environment variables of the form `INTERNAL_TELEMETRY_...` to tweak common options such as detail levels and sampling. Refer to the [configuration itself](https://github.com/newrelic/nrdot-collector-releases/blob/main/examples/internal-telemetry-config.yaml) for more details. We recommend using the default configuration to monitor your collector. However, you can modify the example configuration to meet your needs, such as reducing detail level, sampling or data collection rates. Refer to the [official documentation](https://opentelemetry.io/docs/collector/internal-telemetry/) for more details. Keep in mind that changes to the configuration may cause certain parts of the UI to not display data. Also refer to [Limitations](https://docs.newrelic.com/docs/opentelemetry/collector-observability/collector-observability#limitations). The collector's internal telemetry configuration options change as the OTel community evolves. You have full control over your configuration, including whether to use the provided defaults. ## Overhead considerations [#overhead] Like all telemetry, the collector's internal telemetry increases your data ingest. The overhead depends on your workload and configuration. Here are the key factors to consider: **Metrics** Creates constant overhead for the collector and all active components, regardless of throughput. Metrics are emitted at a constant interval (60 seconds by default). Any component can emit custom metrics, as indicated by the respective [metadata.yaml](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/metadata.yaml#L20), which may add to the overhead. If you need to reduce the amount of metrics your collector emits, we recommend setting the metric level to `normal`, e.g. via the environment variable `INTERNAL_TELEMETRY_METRICS_LEVEL` in the example config, as only a subset of the `detailed` metrics are used by the UI and are typically metrics intended for fine-tuning performance or network issues, so you can re-enable them as needed. **Logs** Has minimal impact during normal operation. Higher overhead can occur when error logs increase rapidly, but this is reduced because logs are sampled by default. The sampling algorithm allows a constant number of logs per defined interval and then samples at a static rate. Refer to [service::telemetry::logs::sampling](https://github.com/newrelic/nrdot-collector-releases/blob/main/examples/internal-telemetry-config.yaml). **Traces** Disabled by default due to: - Limited maturity of traces. Not all components are instrumented, see this [github issue](https://github.com/open-telemetry/opentelemetry-collector/issues/8804). - Unpredictable overhead that varies with workload and batching configuration from upstream agents and collectors. - No adaptive sampling algorithm available. This makes it impossible to provide a universal sampling recommendation that doesn't risk unexpected costs for certain use cases. When traces become more mature, they will provide valuable insight into how much time the collector spends processing data at the component level. To experiment with traces as they develop, set `INTERNAL_TELEMETRY_TRACE_LEVEL=info` and start with a low sampling rate such as `INTERNAL_TELEMETRY_TRACE_SAMPLE_RATIO=0.001` (0.1%) while monitoring your ingest volume and adjusting as needed. ## Configuration examples [#examples] This section contains examples of how the above step-by-step guide translates to different use cases involving the collector. ### Simple collector with internal telemetry enabled [Minimal example](https://github.com/newrelic/newrelic-opentelemetry-examples/tree/main/other-examples/collector/internal-telemetry) of a collector running with docker compose. ### Populate infrastructure relationships (optional, experimental) As mentioned on the [Summary page](https://docs.newrelic.com/docs/opentelemetry/collector-observability/collector-observability#summary-page), APM can show infrastructure metrics for the infrastructure the collector is running on but only if your infrastructure is instrumented and [a relationship](https://docs.newrelic.com/docs/new-relic-solutions/new-relic-one/core-concepts/what-entity-new-relic/#related-entities) can be formed. The relationship is driven by additional attributes on the collector's internal telemetry. You can read more about this topic in our [OTel documentation](https://docs.newrelic.com/docs/opentelemetry/best-practices/opentelemetry-best-practices-resources/#service-to-infra-otel-collector). The steps to set this up heavily depend on your exact infrastructure. For the common choice of Kubernetes we created [an example](https://github.com/newrelic/newrelic-opentelemetry-examples/tree/main/other-examples/collector/internal-telemetry-k8s-container-relationships) that showcases how to achieve container-to-service relationships based on our [OTel for Kubernetes solution](https://docs.newrelic.com/docs/kubernetes-pixie/k8s-otel/intro/).