NVIDIA Triton integration

Install the infrastructure agent

To use the NVIDIA Triton integration, you need to also install the infrastructure agent on the same host. The infrastructure agent monitors the host itself, while the integration you'll install in the next step extends your monitoring with NVIDIA Triton-specific data.

Enable the NVIDIA Triton integration with nri-prometheus

The Triton server metrics are displayed at the URL http://localhost:8002/metrics.

To set up the NVIDIA Triton integration, follow these steps:

1. Run this command to create a file named nri-prometheus-config.yml in the integrations directory:

   bash

   Copy

   $
   touch /etc/newrelic-infra/integrations.d/nri-prometheus-config.yml

2. Add the following snippet to your nri-prometheus-config.yml file to enable the agent to capture NVIDIA Triton data:

   integrations:
     - name: nri-prometheus
       config:
         # When standalone is set to false nri-prometheus requires an infrastructure agent to work and send data. Defaults to true
         standalone: false
   
         # When running with infrastructure agent emitters will have to include infra-sdk
         emitters: infra-sdk
   
         # The name of your cluster. It's important to match other New Relic products to relate the data.
         cluster_name: "YOUR_DESIRED_CLUSTER_NAME"
   
         targets:
           - description: NVIDIA Triton metrics list
             urls: ["http://localhost:8002/metrics"]
   
         #     tls_config:
         #     ca_file_path: "/etc/etcd/etcd-client-ca.crt"
         #     cert_file_path: "/etc/etcd/etcd-client.crt"
         #      key_file_path: "/etc/etcd/etcd-client.key"
   
         # Whether the integration should run in verbose mode or not. Defaults to false
         verbose: false
   
         # Whether the integration should run in audit mode or not. Defaults to false.
         # Audit mode logs the uncompressed data sent to New Relic. Use this to log all data sent.
         # It does not include verbose mode. This can lead to a high log volume, use with care
         audit: false
   
         # The HTTP client timeout when fetching data from endpoints. Defaults to 30s.
         # scrape_timeout: "30s"
   
         # Length in time to distribute the scraping from the endpoints
         scrape_duration: "5s"
   
         # Number of worker threads used for scraping targets.
         # For large clusters with many (>400) endpoints, slowly increase until scrape
         # time falls between the desired `scrape_duration`.
         # Increasing this value too much will result in huge memory consumption if too
         # many metrics are being scraped.
         # Default: 4
         # worker_threads: 4
   
         # Whether the integration should skip TLS verification or not. Defaults to false
         insecure_skip_verify: true
       timeout: 10s

   Copy

NVIDIA Triton logs configuration

To configure nvidia triton logs, follow the steps outlined below.

1. Run this Docker command to check the status of running containers:

   bash

   Copy

   $
   sudo docker ps

2. Copy the container ID for the nvidia-triton container and execute this command:

   bash

   Copy

   $
   sudo docker logs -f <container_id> &> /tmp/triton.log &

Afterwards, verify there is a log file named triton.log located in the /tmp/ directory.

Forwarding NVIDIA Triton logs to New Relic

You can use our log forwarding to forward NVIDIA Triton logs to New Relic. On Linux machines, your log file named logging.yml should be in this path:

$
cd /etc/newrelic-infra/logging.d/

Once you find the log file in the above path, include this script into the logging.yml file:

logs:
  - name: triton.log
    file: /tmp/triton.log
    attributes:
      logtype: triton_logs

Restart the New Relic infrastructure agent

Run this command to restart your infrastructure agent:

$
sudo systemctl restart newrelic-infra.service

In a couple of minutes, your NVIDIA Triton server will send metrics to one.newrelic.com.

Find your data

You can choose our pre-built dashboard template named NVIDIA Triton to monitor your NVIDIA Triton server metrics. Follow these steps to use our pre-built dashboard template:

1. Go to one.newrelic.com > + Add data and type NVIDIA Triton.
2. Under Dashboards, click NVIDIA Triton.
3. Click Edit if you want to change the account in the open popup window.
4. Click Setup NVIDIA Triton or Skip this step if you already setup this datasource.
5. Click View dashboard, and see your NVIDIA Triton data in New Relic.

You can find your custom NVIDIA Triton dashboard in the Dashboards UI. See our dashboard section for more information.

Here is a NRQL query to check the NVIDIA Triton CPU memory:

SELECT latest(nv_cpu_memory_total_bytes) / 1e+6 AS 'memory (MB)' FROM Metric