• English日本語한국어
  • Log inStart now

AWS Lambda for sending logs from S3

You can send your Amazon S3 buckets to New Relic using our AWS Lambda function, NewRelic-log-ingestion-s3. This can be easily deployed from the AWS Serverless application repository.

Forwarding logs from your S3 bucket to New Relic will give you enhanced capabilities to collect, process, explore, query, and alert on your log data.

Install the Lambda function

To install the Lambda function to forward your S3 logs to New Relic:

  1. Make sure you have a .
  2. Open the AWS Serverless Application Repository in your browser.
  3. Make sure that the Lambda is installed in the same region as the S3 bucket.
  4. Search for newrelic and check Show apps that create custom IAM roles or resource policies to find NewRelic-log-ingestion-s3.
  5. Click the NewRelic-log-ingestion-s3 details and click Deploy.
  6. Scroll to the Application settings and enter your New Relic .
  7. Take advantage of New Relic's log parsing capabilities by specifying the logtype as an environment variable for the Lambda function. For more information, see Built-in parsing rulesets.
  8. Confirm that the app creates custom IAM roles, and then click Deploy.
  9. Once the function is deployed, create a Lambda trigger.

Manual install using Serverless Framework

You can manually install and configure the Lambda from the terminal using the Serverless Framework. If you're not on Linux, install Docker first and keep it running.

bash
$
# Clone the repository
$
git clone https://github.com/newrelic/aws_s3_log_ingestion_lambda.git
$
# Install <a href="https://www.npmjs.com/package/serverless">Serverless</a> and the repo dependencies
$
npm install -g serverless
$
npm install
$
# Set the New Relic License Key env variable
$
export LICENSE_KEY=YOUR_NEWRELIC_LICENSE_KEY
$
# Set the logtype (optional)
$
export LOG_TYPE=YOUR_LOGTYPE
$
# Set the S3 bucket name
$
export S3_BUCKET_NAME=YOUR_S3_BUCKET_NAME
$
# The S3 subdirectory (optional)
$
export S3_PREFIX=YOUR_S3_SUBDIRECTORY
$
# Choose a name for your new Serverless Framework service
$
export SERVICE_NAME=YOUR_SERVICE_NAME
$
# Deploy the function, specifying the same region as your S3 bucket listed above
$
serverless deploy --region YOUR_AWS_REGION

Create a Lambda trigger in AWS

To get your logs streaming to New Relic, attach a trigger to the Lambda:

  1. In the left side menu, click Functions.
  2. Find and select the previously created NewRelic-s3-log-ingestion function.
  3. Under Designer, click Add Triggers and select S3 from the dropdown.
  4. Select the S3 bucket that contains the log you want to send to New Relic.
  5. Optionally configure a prefix and suffix. This is helpful if your logs are in a subdirectory.
  6. Check the Enable trigger checkbox, then Add the trigger.

To test the Lambda function, upload a new log file to the S3 bucket you configured in the function settings and then view your log data.

Did this doc help with your installation?

Limitations

  • The maximum supported log file size is 35 MB uncompressed.
  • The maximum supported individual log line size is 0.25 MB.
  • Encrypted log files are not supported.

View log data

If everything is configured correctly and your data is being collected, you should see log data in both of these places:

For example, you can execute a query like this:

SELECT * FROM Log

Troubleshooting

If you encounter problems with configuring your log forwarder, try these troubleshooting tips.

What's next?

Explore logging data across your platform.

Disable log forwarding

To disable log forwarding capabilities, follow standard procedures in Amazon S3 documentation. You do not need to do anything else in New Relic.

Copyright © 2024 New Relic Inc.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.