Monitor and troubleshoot ATP

• To view recent process metrics:

  FROM Metric SELECT *
  WHERE metricName LIKE 'process.%'
  LIMIT 100

  コピー

• To count unique process metric types:

  FROM Metric SELECT uniqueCount(metricName)
  WHERE metricName LIKE 'process.%'

  コピー

• To view recent process metrics for your cluster:

  FROM Metric SELECT *
  WHERE k8s.cluster.name='your-cluster-name'
  AND metricName LIKE 'process.%'
  LIMIT 100

  コピー

• To count unique process metric types for your cluster:

  FROM Metric SELECT uniqueCount(metricName)
  WHERE k8s.cluster.name='your-cluster-name'
  AND metricName LIKE 'process.%'

  コピー

Problem: All process metrics are being sent, not filtered.

Problem: Collector memory increases over time

Solution: Adjust retention settings:

processors:
  adaptivetelemetry:
    retention_minutes: 15  # Reduce from default 30

Problem: Expected processes are missing from telemetry

Solution: Must use full path to add processes to include list. Entries without path separators won't match any process. For example:

processors:
  adaptivetelemetry:
    include_process_list:
      - "/usr/bin/myapp"  # Always captured regardless of thresholds

Problem: Earlier, ATP data was shown but now it is not. This could be due to hitting cardinality limits. ATP filters out many processes to reduce telemetry volume, but if you have a very high number of unique processes, you could still hit limits. You can monitor your cardinality usage to track this.

Solution:

To troubleshoot, check whether you've hit cardinality limits by going to the cardinality management page in New Relic. If you've breached limits, you can increase your cardinality limit.

processors:
  adaptivetelemetry:
    metric_thresholds:
      process.cpu.utilization: 0.10  # Increase from 0.05 to 0.10 (10% CPU)
      process.memory.utilization: 0.10  # Filter out more processes

To enable ATP's persistence file feature, you need to run the binary with escalated privileges.

Add the following to the daemonset.yaml file in your Helm chart:

hostPID: true

To enable disk.io data collection by the OpenTelemetry hostmetrics receiver, you need escalated privileges because the io data stored at /proc/[PID]/io requires root user access. This is not an ATP requirement, ATP only processes data that's already been collected by OpenTelemetry.

Add the following security context to your Helm chart:

securityContext:
  allowPrivilegeEscalation: true
  privileged: true
  readOnlyRootFilesystem: false
  runAsNonRoot: false
  runAsUser: 0