Troubleshoot eBPF on Kubernetes

Problem

You're experiencing issues with the eBPF agent in your Kubernetes environment, such as file descriptor limits, privilege errors, or performance problems.

Solution

If you encounter file descriptor limit errors, see our dedicated file descriptor limit troubleshooting guide for detailed resolution steps.

Problem

eBPF agent fails to start due to insufficient privileges.

Solution

1. Verify that the eBPF agent DaemonSet has the necessary privileges. The Helm chart should automatically configure required permissions.

2. Check pod security context in your deployment:

   bash

   카피

   $
   kubectl describe pod <ebpf-agent-pod> -n newrelic

3. Ensure your cluster supports eBPF. Check kernel version on nodes:

   bash

   카피

   $
   kubectl get nodes -o wide
   $
   # Kernel version should be 5.4 or later

Problem

eBPF agent consuming excessive resources or causing performance degradation.

Solution:

1. Monitor resource usage:

   bash

   카피

   $
   kubectl top pods -n newrelic

2. Adjust memory limits in your Helm values:

   agent:
   resources:
      limits:
         memory: "2Gi"  # Increase if needed
      requests:
         memory: "512Mi"

   카피

3. Configure data filtering to reduce load:

   dropDataForNamespaces: ["kube-system", "monitoring"]
   dropDataServiceNameRegex: "kube-dns|otel-collector"

   카피

4. Limit protocol monitoring if not all protocols are needed:

   protocols:
   http:
      enabled: true
   mysql:
      enabled: false  # Disable if not needed

   카피

Problem

eBPF agent pods not starting or unable to send data.

Solution

1. Check pod status:

   bash

   카피

   $
   kubectl get pods -n newrelic
   $
   kubectl describe pod <ebpf-agent-pod> -n newrelic

2. Review pod logs:

   bash

   카피

   $
   kubectl logs <ebpf-agent-pod> -n newrelic

3. Verify network connectivity:

   bash

   카피

   $
   # Test from within the cluster
   $
   kubectl run test-connectivity --image=busybox --rm -it --restart=Never -- \
   >
   nslookup otlp.nr-data.net

4. Check service account and RBAC:

   bash

   카피

   $
   kubectl get serviceaccount -n newrelic
   $
   kubectl get clusterrole,clusterrolebinding -l app.kubernetes.io/name=nr-ebpf-agent

Problem

Entity names not appearing correctly or data not attributed to correct services.

Solution

• The eBPF agent uses Kubernetes Service objects to name entities. Ensure your applications have a corresponding service defined.

  apiVersion: v1
  kind: Service
  metadata:
  name: my-service  # This becomes the entity name
  spec:
  selector:
     app: my-app

  카피

• If you are missing data, ensure the namespace is not being excluded in your values.yaml.

  # In values.yaml
  dropDataForNamespaces: []  # Remove namespaces you want to monitor

  카피

• In Kubernetes, entity names are derived from the Kubernetes service name for example, mysql-database-service. On hosts or in Docker, names are a combination of the process name, its directory or container ID, and the listening port for example, ruby:/home/ubuntu/app:[5678].

Verification steps

1. Check for successful startup log:

   bash

   카피

   $
   kubectl logs <ebpf-agent-pod> -n newrelic | grep "STEP-7"

   Should show: [STEP-7] => Successfully started the eBPF Agent.

2. Verify data flow in New Relic:

   • In the New Relic UI, look for entities with instrumentation.name = nr_ebpf.
   • Confirm that the entity names match your Kubernetes service names.

3. Test OTLP endpoint connectivity:

   bash

   카피

   $
   kubectl exec -it <ebpf-agent-pod> -n newrelic -- \
   >
   curl -v https://otlp.nr-data.net:443