NGINX distributed tracing with OpenTelemetry

Load the module and enable tracing in your nginx.conf. Add the load_module directive at the top level (main context), and the OpenTelemetry directives inside the http context:

# Main context — load the dynamic module
load_module modules/ngx_otel_module.so;

http {
    # Export spans to the OpenTelemetry Collector over OTLP/gRPC
    otel_exporter {
        endpoint localhost:4317;
    }

    # A name that identifies this NGINX instance in traces
    otel_service_name nginx-server;

    # Emit a span per request and propagate W3C trace context to upstreams
    otel_trace on;
    otel_trace_context propagate;

    server {
        listen 80;

        location / {
            proxy_pass http://backend_app;
        }
    }
}

This configuration:

• Loads the OpenTelemetry module (load_module).
• Exports spans over OTLP/gRPC to the collector listening on localhost:4317 (otel_exporter).
• Creates a span for every request (otel_trace on). To trace a subset of traffic in high-throughput environments, set otel_trace to a variable (for example, driven by split_clients) instead of on.
• Propagates trace context (otel_trace_context propagate): This both extracts the incoming traceparent header (linking NGINX spans to the calling service) and injects updated context into requests sent to upstreams (letting downstream services continue the trace).

The otel_trace on and otel_trace_context directives can also be set per server or location block if you want to trace only specific virtual hosts or routes.

For the full directive reference, see the ngx_otel_module documentation.

Configure the OTel Collector to receive traces from ngx_otel_module and metrics from the NGINX stub status endpoint, enrich both with the NGINX identity, and forward them to New Relic.

receivers:
  # Receives spans from ngx_otel_module (NGINX → localhost:4317)
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"
  # Collects NGINX metrics from the stub status endpoint
  nginx:
    endpoint: <YOUR_STUB_STATUS_ENDPOINT>  # e.g. http://127.0.0.1/status
    collection_interval: 30s

processors:
  resourcedetection:
    detectors: [system]
    system:
      resource_attributes:
        host.id:
          enabled: true

  # Adds the NGINX identity so spans and metrics resolve to the SAME NGINXSERVER entity
  resource/nginx:
    attributes:
      - key: nginx.server.endpoint
        value: "<YOUR_STUB_STATUS_ENDPOINT>"  # must match the nginx receiver endpoint
        action: upsert
      - key: nginx.deployment.name
        value: "<YOUR_DEPLOYMENT_NAME>"        # a stable name for this NGINX deployment
        action: upsert

  # Sets nginx.display.name for the metrics pipeline
  transform/nginx_metrics:
    metric_statements:
      - context: resource
        statements:
          - set(attributes["nginx.display.name"], Concat(["server", attributes["nginx.deployment.name"]], ":"))

  # Same display-name stamp for the traces pipeline — ngx_otel_module spans carry no
  # NGINX identity by default; resource/nginx adds the endpoint + deployment name, and
  # this adds nginx.display.name, so the spans resolve to the NGINXSERVER entity.
  transform/nginx_traces:
    trace_statements:
      - context: resource
        statements:
          - set(attributes["nginx.display.name"], Concat(["server", attributes["nginx.deployment.name"]], ":"))

  batch:

exporters:
  otlp_http:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    headers:
      api-key: ${env:NEW_RELIC_LICENSE_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resourcedetection, resource/nginx, transform/nginx_traces, batch]
      exporters: [otlp_http]
    metrics/nginx:
      receivers: [nginx]
      processors: [resourcedetection, resource/nginx, transform/nginx_metrics, batch]
      exporters: [otlp_http]

This collector configuration includes two pipelines:

• Traces pipeline: Receives OTLP trace data from ngx_otel_module and your instrumented applications via gRPC (port 4317) or HTTP (port 4318). This is the same standard traces pipeline you would use for any service sending OTLP data to New Relic.
• Metrics pipeline: Uses the nginxreceiver to collect performance metrics (connections, requests) from the NGINX stub status endpoint. These metrics create the NGINX entity in New Relic with golden metrics.

Both pipelines share these processors:

• resourcedetection: Adds host.id, a standard resource attribute used to identify hosts across the OpenTelemetry ecosystem.
• resource/nginx: Adds nginx.server.endpoint and nginx.deployment.name. These two attributes form the NGINX entity's identity. Applying them to both pipelines makes the NGINX spans and metrics resolve to the same NGINXSERVER entity instead of a duplicate service entity.
• transform/nginx_*: Adds nginx.display.name for a friendly entity name.

Set the required environment variables and start (or restart) the collector:

$
export NEW_RELIC_LICENSE_KEY="<YOUR_LICENSE_KEY>"
$
export OTEL_EXPORTER_OTLP_ENDPOINT="<YOUR_NEWRELIC_OTLP_ENDPOINT>"
$

$
sudo systemctl restart nrdot-collector

Replace <YOUR_LICENSE_KEY> with your ライセンスキー. For the OTLP endpoint, refer to New Relic OTLP endpoint configuration.

Test the configuration and reload NGINX to load the OpenTelemetry module:

$
sudo nginx -t
$
sudo systemctl reload nginx

Once NGINX and the collector are running, generate some traffic through your instrumented applications. After a few minutes, verify data is arriving in New Relic:

-- Verify NGINX trace spans
FROM Span SELECT count(*)
WHERE nginx.deployment.name = '<YOUR_DEPLOYMENT_NAME>'
SINCE 10 minutes ago

-- Verify NGINX metrics
FROM Metric SELECT count(*)
WHERE metricName LIKE 'nginx.%'
SINCE 10 minutes ago

After trace data is flowing, New Relic automatically creates CALLS relationships between your services and the NGINX entity. It may take up to 10 minutes for relationships to appear.

To view the relationships:

1. Go to one.newrelic.com > All capabilities > All entities.
2. Search for your NGINX entity or one of your instrumented services.
3. Select an entity to open its summary page.
4. Click Service map to see the entity relationship graph.

You should see your client services connected to NGINX, and NGINX connected to your backend services:

[Client app] → CALLS → [NGINX] → CALLS → [Backend app]

You can also query relationships with NRQL:

FROM Relationship SELECT *
WHERE source.entityName = '<YOUR_NGINX_DISPLAY_NAME>'
  OR target.entityName = '<YOUR_NGINX_DISPLAY_NAME>'
SINCE 1 day ago

• Verify NGINX reloaded without errors: sudo nginx -t and sudo journalctl -u nginx -n 50 --no-pager
• Confirm ngx_otel_module is loaded. An unknown directive "otel_exporter" error means the module isn't loaded. Check the load_module path and that nginx-module-otel is installed for your NGINX version.
• Verify the OTel Collector is running and listening on the port in otel_exporter: sudo ss -tlnp | grep 4317
• Check collector logs for errors: sudo journalctl -u nrdot-collector -n 50 --no-pager
• Confirm the otel_exporter endpoint matches the collector's gRPC listener address.

• Allow up to 10 minutes for relationships to appear after the first spans arrive.
• Verify that your instrumented applications are sending traces through the collector. Both the NGINX spans and the application spans must reach New Relic for relationships to form.
• Check that your client applications propagate W3C traceparent headers. Without trace context propagation, NGINX spans aren't connected to the calling service.
• Confirm both the resourcedetection and resource/nginx processors are included in the collector traces pipeline. The nginx.server.endpoint and nginx.deployment.name attributes are required for the NGINX spans to resolve to the NGINXSERVER entity.
• Confirm otel_trace_context propagate is set (not extract or inject alone). propagate is required for end-to-end context flow through NGINX.
• Query to verify both NGINX and application spans share trace IDs:

  FROM Span SELECT uniques(service.name)
  WHERE trace.id IN (
    SELECT uniques(trace.id) FROM Span
    WHERE nginx.deployment.name = '<YOUR_DEPLOYMENT_NAME>'
    SINCE 10 minutes ago LIMIT 5
  )
  SINCE 10 minutes ago

  コピー
  You should see your NGINX service alongside your application service names.

• Ensure the resource/nginx processor runs in the traces pipeline (not only the metrics pipeline). Without it, NGINX spans lack nginx.deployment.name / nginx.server.endpoint and are synthesized as a generic service rather than resolving to the NGINXSERVER entity.
• Verify the nginx.server.endpoint value is identical in the nginx receiver and the resource/nginx processor, and that nginx.deployment.name matches the value used by your NGINX metrics. The entity identity is the composite of these two values. A mismatch produces a different entity.
• Run the metrics pipeline too. The NGINXSERVER entity's golden metrics come from the nginxreceiver; without it, NGINX still appears via traces but without metrics.

• Each instrumented application must send traces through the same OTel Collector (or directly to New Relic) so that span data for all services reaches the same account.
• For applications using New Relic APM agents, verify that distributed tracing is enabled and the agent is connected.
• For OTel SDK applications, verify the OTLP exporter is configured to send to the collector.
• Allow additional time. Relationships for services with lower traffic volume may take longer to appear.