• Log inStart now

Stream logs using Kinesis Data Firehose

If your log data is already being monitored by Amazon CloudWatch Logs, you can use our Kinesis Data Firehose integration to forward and enrich your log data in New Relic. Kinesis Data Firehose is a service that can stream data in real time to a variety of destinations, including our platform.

Forwarding your CloudWatch Logs or other logs compatible with a Kinesis stream to New Relic will give you enhanced log management capabilities to collect, process, explore, query, and alert on your log data.

Create the delivery stream for New Relic

To forward your logs from Kinesis Data Firehose to New Relic:

  1. Make sure you have a New Relic license key.

  2. Follow the procedures to create an Amazon Kinesis Data Firehose delivery stream, and select New Relic as the destination: In the AWS Management Console, go to Amazon Kinesis.

  3. Select Kinesis Data Firehose and click Create delivery stream.

  4. Enter a name for the stream and select your data source.

  5. When selecting the destination, click Third-party partner and then New Relic.

  6. Under HTTP endpoint URL, select New Relic logs - US. This is the US endpoint (https://aws-api.newrelic.com/firehose/v1). To use the EU endpoint, complete the remaining steps in this procedure, then go to EU account configuration.

  7. Paste your license key in the API key field.

  8. Ensure that Content encoding is set to GZIP.

  9. Set a buffer size of 1 MiB under Buffer hints/Buffer size. (Note that the Buffer hints section is folded by default. For more information, see the buffer settings section.

  10. Configure and review the remaining metadata settings.

Any optional key/value pairs you add in the AWS Management Console will result in attribute/value pairs you can use in New Relic. Kinesis Data Firehose includes these key-value pairs in each HTTP call. These Kinesis Firehose parameters can help you identify and organize your destinations.

Configure buffer size and interval

When selecting our Kinesis Firehose integration for logs in AWS, the wizard hides a small section called Buffer hints at the bottom of the screen. This section contains two very important configuration options: Buffer size and Buffer interval.

The default values are 5 MiB for Buffer size, and 60 seconds for Buffer interval. This means that Firehose accumulates logs until either they contain 5 MiB of data, or until 60 seconds have passed since the last time they were flushed to New Relic.

Important

These default settings are not appropriate for log management in New Relic. We strongly advise to use 1 MiB as the Buffer size and activate GZIP body compression.

Why do we recommend this? New Relic accepts at most payloads of 1 MB (1.000.000 bytes) for an individual HTTP POST request to its Logs API. If the size of the accumulated logs exceeds 1 MB during any 60-second accumulation period, we will reject those logs with a 413 HTTP error.

The delivery stream allows configuring, at a minimum, a buffer size of 1 MiB (1.048.576 bytes). Even though this value is slightly greater than the 1MB allowed by New Relic's Logs API, you can use GZIP compression so that the resulting payload becomes smaller than the 1MB limit.

Configure your stream to send logs to EU accounts

If you have a New Relic account in our EU data center:

  1. Follow standard procedures to create your delivery stream.

  2. Once your stream has been created, navigate to the Amazon Kinesis Console, and click Delivery streams.

  3. Click the name of your newly created delivery stream, and select the Configuration tab.

  4. Next to the Destination settings section, click Edit.

  5. Update the value for HTTP endpoint URL to our Firehose endpoint for EU accounts:

    https://aws-api.eu.newrelic.com/firehose/v1
  6. Click Save changes to ensure your stream is updated.

While you wait for your data to come in...

Did this doc help you install?

View log data

If everything is configured correctly and your data is being collected, you should see data logs in both of these places:

SELECT * FROM Log

If no data appears after you enable our log management capabilities, follow our standard log troubleshooting procedures.

What's next?

Explore logging data across your platform with our Logs UI.

Disable log forwarding

To disable log forwarding capabilities, follow standard procedures in Kinesis Data Firehose documentation. You do not need to do anything else in New Relic.

Copyright © 2022 New Relic Inc.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.