If your log data is already being monitored by Amazon CloudWatch Logs, you can use our Kinesis Data Firehose integration to forward and enrich your log data in New Relic. Kinesis Data Firehose is a service that can stream data in real time to a variety of destinations, including our platform.
Forwarding your CloudWatch Logs or other logs compatible with a Kinesis stream to New Relic will give you enhanced log management capabilities to collect, process, explore, query, and alert on your log data.
To forward your logs from Kinesis Data Firehose to New Relic:
Make sure you have a New Relic license key.
Follow the procedures to create an Amazon Kinesis Data Firehose delivery stream, and select New Relic as the destination: In the AWS Management Console, go to Amazon Kinesis.
Select Kinesis Data Firehose and click Create delivery stream.
Enter a name for the stream and select your data source.
When selecting the destination, click Third-party partner and then New Relic.
Under HTTP endpoint URL, select New Relic logs - US. This is the US endpoint (https://aws-api.newrelic.com/firehose/v1). To use the EU endpoint, complete the remaining steps in this procedure, then go to EU account configuration.
Paste your license key in the API key field.
Ensure that Content encoding is set to
Set a buffer size of
Buffer hints/Buffer size. (Note that the
Buffer hintssection is folded by default. For more information, see the buffer settings section.
Configure and review the remaining metadata settings.
Any optional key/value pairs you add in the AWS Management Console will result in attribute/value pairs you can use in New Relic. Kinesis Data Firehose includes these key-value pairs in each HTTP call. These Kinesis Firehose parameters can help you identify and organize your destinations.
When selecting our Kinesis Firehose integration for logs in AWS, the wizard hides a small section called
Buffer hints at the bottom of the screen. This section contains two very important configuration options:
Buffer size and
The default values are 5 MiB for
Buffer size, and 60 seconds for
Buffer interval. This means that Firehose accumulates logs until either they contain 5 MiB of data, or until 60 seconds have passed since the last time they were flushed to New Relic.
These default settings are not appropriate for log management in New Relic. We strongly advise to use 1 MiB as the
Buffer size and activate GZIP body compression.
Why do we recommend this? New Relic accepts at most payloads of 1 MB (1.000.000 bytes) for an individual HTTP POST request to its Logs API. If the size of the accumulated logs exceeds 1 MB during any 60-second accumulation period, we will reject those logs with a
413 HTTP error.
The delivery stream allows configuring, at a minimum, a buffer size of 1 MiB (1.048.576 bytes). Even though this value is slightly greater than the 1MB allowed by New Relic's Logs API, you can use GZIP compression so that the resulting payload becomes smaller than the 1MB limit.
If you have a New Relic account in our EU data center:
Follow standard procedures to create your delivery stream.
Once your stream has been created, navigate to the Amazon Kinesis Console, and click Delivery streams.
Click the name of your newly created delivery stream, and select the Configuration tab.
Next to the Destination settings section, click Edit.
Update the value for HTTP endpoint URL to our Firehose endpoint for EU accounts:https://aws-api.eu.newrelic.com/firehose/v1
Click Save changes to ensure your stream is updated.
If everything is configured correctly and your data is being collected, you should see data logs in both of these places:
- New Relic's Logs UI
- New Relic tools for running NRQL queries. For example, you can execute a query like this:
SELECT * FROM Log
If no data appears after you enable our log management capabilities, follow our standard log troubleshooting procedures.
Explore logging data across your platform with our Logs UI.
- Get deeper visibility into both your application and your platform performance data by forwarding your logs with our logs in context capabilities.
- Set up alerts.
- Query your data and create dashboards.
To disable log forwarding capabilities, follow standard procedures in Kinesis Data Firehose documentation. You do not need to do anything else in New Relic.