Troubleshoot Metric API with NRIntegrationError events


You sent metric data points to the Metric API, and are not seeing what you expect when querying the data. Use the following checklist to determine the root cause:

  • Make sure you are querying the data correctly.
  • Check the HTTP status codes returned by the API. Issues like authorization failures can be diagnosed with HTTP status codes.
  • If you are sending data from a Prometheus server via New Relic's remote_write endpoint, check your Prometheus server logs for errors or non-2xx HTTP responses from the New Relic endpoint.
  • Query your account for NrIntegrationError events. New Relic's ingestion endpoints are asynchronous, meaning the endpoint verifies the payload after it returns the HTTP response. If any issues occur while verifying your payload, then an NrIntegrationError event will be created in your account. New Relic also uses NrIntegrationError events to notify customers when various rate limits have been reached.


View error details

To view details about the errors, run this NRQL query:

SELECT count(*) FROM NrIntegrationError WHERE newRelicFeature ='Metrics' facet category, message limit 100 since 24 hours ago 

The category indicates the type of error and the message provides more detailed information about the error. If the category is rateLimit, then you should also examine the rateLimitType field for more information on the type of rate limiting.

Category rateLimitType Description and solution
BadRequest (not set)

There is an issue with the JSON payload. These include JSON syntax errors, attribute names, or values that are too long.

Check the message field to determine the exact issue. Then review the JSON payload, and update it to ensure it meets the proper semantic guidelines.

RateLimit DatapointsPerMinute You are sending too many datapoints per minute. If you get this error, you can either send data less frequently, or request changes to your metric rate limits by contacting your New Relic account representative, or visiting our Support portal.
RateLimit UniqueTimeseriesPerDay

You have an attribute with a high number of unique values, like containerId or URI. To resolve this error, review any attributes that may be causing the issue and remove them. If desired, you can use a data dropping rule to remove attributes at ingest time.

RateLimit UniquePrometheusTimeseries

You have Prometheus servers reporting too many unique timeseries via New Relic's remote_write endpoint.

Reduce the number of unique timeseries reported by modifying your Prometheus server configuration to reduce the number of targets being scraped, or by using relabel rules in the remote_write section of your server configuration to drop timeseries or highly unique labels.

RateLimit RequestsPerMinute Too many requests per minute are being sent. To resolve this, put more datapoints in each request, and send them less frequently.

Match errors to ingested payloads

When a NrIntegrationError event is created as a result of a syntax issue with the HTTP request payload, then the event contains the attributes apiKeyPrefix and requestId.

  • The apiKeyPrefix matches the first 6 characters of the API key used to send the data.
  • The requestId matches the requestId sent in the HTTP response.

To view these fields, run this NRQL query:

SELECT message, apiKeyPrefix, requestId FROM NrIntegrationError limit 100

To verify a specific requestId, run this NRQL query:

SELECT * FROM NrIntegrationError where requestId ='REQUEST_ID'

Programmatically retrieve NrIntegrationError events

To programmatically retrieve these errors:

  1. Ensure you have an Insights query API key (go to > Manage data > API keys).
  2. Create an HTTP request as shown below:

    If your account hosts data in the EU data center, ensure you're using the proper API endpoints for EU region accounts.

    curl -H "Accept: application/json" -H "X-Query-Key:YOUR_API_KEY_HERE" "*%20FROM%20NrIntegrationError%20where%20newRelicFeature='Metrics'"

For more help

If you need more help, check out these support and learning resources: