Troubleshoot Linkerd monitoring with OpenTelemetry

Verify that Linkerd proxy metrics are reaching New Relic:

SELECT count(*) FROM Metric
WHERE linkerd_control_plane_ns IS NOT NULL
  AND k8s.cluster.name = '<YOUR_CLUSTER>'
SINCE 5 minutes ago

If this returns 0, check that the collector pods are running and the Prometheus scrape jobs in the collector config are targeting the correct namespaces.

Workload entities require kube-state-metrics. Verify it is sending data:

SELECT uniques(k8s.deployment.name)
FROM Metric
WHERE metricName LIKE 'kube_deployment%'
  AND k8s.cluster.name = '<YOUR_CLUSTER>'
SINCE 10 minutes ago

If empty, confirm kube-state-metrics is running and the scrape target in the collector config points to the correct namespace and service name.

$
# Verify Java agent loaded (for APM-instrumented app pods)
$
kubectl logs -n <YOUR_NAMESPACE> deployment/<YOUR_APP> | grep "javaagent"
$

$
# Verify Linkerd proxy routing for OTLP gRPC (must show appProtocol: grpc)
$
kubectl get svc -n newrelic my-opentelemetry-collector -o jsonpath='{.spec.ports}'

The collector Service port 4317 must have appProtocol: grpc. Without it, Linkerd routes gRPC as HTTP/1.1. Fix:

$
kubectl patch svc -n newrelic my-opentelemetry-collector --type='json' \
>
  -p='[{"op":"add","path":"/spec/ports/1/appProtocol","value":"grpc"}]'

The k8sattributes/traces processor uses k8s.pod.ip for pod association. Verify the downward API env var is set on your app pods:

$
kubectl exec -n <YOUR_NAMESPACE> deployment/<YOUR_APP> -- env | grep MY_POD_IP

If MY_POD_IP is missing, add it to your pod spec as a fieldRef env var pointing to status.podIP.

The filelog receiver reads log files under the varlogpods hostPath mount, filtered down to the linkerd-proxy container (OTel Collector Contrib) or every container (NRDOT). Verify the mount is present and readable:

$
# OTel Collector Contrib (separate DaemonSet release for logs)
$
kubectl exec -n newrelic daemonset/my-opentelemetry-collector-logs -- ls /var/log/pods 2>/dev/null | head -5
$

$
# NRDOT (built-in, same DaemonSet as metrics/traces)
$
kubectl exec -n newrelic daemonset/nr-k8s-otel-collector-daemonset -- ls /var/log/pods 2>/dev/null | head -5

If the mount is readable but no linkerd-proxy log records show up in New Relic (OTel Collector Contrib), check that filter/onlyproxy's k8s.container.name != "linkerd-proxy" condition matches your pods. add_metadata_from_filepath: true on the filelog receiver's container operator is what populates k8s.container.name from the log file path.