Enable New Relic Logs for AWS CloudWatch

New Relic Logs offers an integrated plugin with AWS CloudWatch monitoring, allowing you to send your CloudWatch Logs to New Relic Logs.

Compatibility and requirements

To use New Relic Logs with the AWS Serverless Application plugin, ensure your configuration meets the following requirements:

Enable AWS Serverless Application for New Relic Logs

To enable New Relic Logs with the AWS plugin:

  1. Install the New Relic Lambda
  2. Configure the Lambda
  3. Create a Lambda trigger
  4. Configure retries
  5. Generate some traffic and wait a few minutes, then check your account for data.

Install the New Relic Lambda

To install the New Relic Lambda, use the AWS Serverless Application Repository, which is where the newrelic-log-ingestion Lambda is stored. This repo is a collection of serverless applications published by developers, companies, and partners in the serverless community. It allows developers to share their Lambda functions code with customers, who can then find and deploy the corresponding application Lambda function. Each application is packaged with an AWS Serverless Application Model (SAM) template that defines the AWS resources used.

To create the New Relic Lambda with the AWS Serverless Application Repository:

  1. From the AWS console, go to the Lambda section, select Create function, and select Serverless Application Repository.
  2. Search for newrelic and check the box that says "Show apps that create custom IAM roles or resource policies" to find the newrelic-log-ingestion Lambda. Follow the instructions in the Lambda's documentation to deploy it. A SAM template will build the Lambda.

Configure your Lambda function

You can configure forwarding of logs to New Relic Infrastructure and/or New Relic Logging using the following environment variables:

Key Description Value
LICENSE_KEY New Relic License key, needed for sending data to both Infrastructure and Logging. If set, enables forwarding logs to New Relic Infrastructure by default. Required. Your New Relic license key.
LOGGING_ENABLED Determines if logs are forwarded to New Relic Logging. Required. Default: false
INFRA_ENABLED Determines if logs are forwarded to New Relic Infrastructure. Required. Default: false
NR_LOGGING_ENDPOINT New Relic ingestion endpoint for Logging. Optional. US: https://log-api.newrelic.com/log/v1
EU: http://log-api.eu.newrelic.com/log/v1
NR_INFRA_ENDPOINT New Relic ingestion endpoint for Infrastructure. Optional. https://cloud-collector.newrelic.com
  • Once you have updated the function.py and added the environment variables, click Save. Once complete, link your Lambda function to CloudWatch Logs.

Create a Lambda trigger

To get your logs streaming to New Relic you will need to attach a trigger to the Lambda:

  1. From the left side menu, select Functions.
  2. Find and select the previously created newrelic-log-ingestion function.
  3. Under Designer, click Add Triggers, and select Cloudwatch Logs from the dropdown.
  4. Select the the appropriate Log group for your application.
  5. Enter a name for your filter.
  6. Optional: Enter a filter pattern.
  7. Check the Enable trigger checkbox, then click Add to create the trigger.

Configure retries (optional)

You can configure the number of retries you want to perform in case the function fails to send the data in case of communication issues. Recommended number is 3 retries, but you can change the retry behavior by changing the below parameters:

Please be aware that more number of retries can make the function run for longer time and therefore increases the probability of having higher costs for Lambda. On the contrary, decreasing the number of retries could increase the probability of data loss.

MAX_RETRIES = 3  # Defines the number of retries after lambda failure to deliver data
INITIAL_BACKOFF = 1  # Defines the initial wait seconds until next retry is executed
BACKOFF_MULTIPLIER = 2  # Time multiplier between the retries
As an example, in default above configuration, first retry will happen after 1 second, second retry after 2 seconds and third retry will happen after 4 seconds.

Resources created by the SAM template

There are few resources that will be created when you create the application from the repository:

  • The Lambda function itself
  • A Role used to give execution permissions to the Lambda function based in CloudWatch Logs.

All other lambda configurations not listed in the steps above can be left as the defaults.

View log data

If everything is configured correctly and your data is being collected, you should see data logs in the New Relic Logs UI or by going to Insights and querying:

SELECT * FROM Log

For more help

Recommendations for learning more: