AWS Lambda for sending logs from S3

You can send your logs from Amazon S3 buckets to New Relic using our AWS Lambda function, NewRelic-log-ingestion-s3, which can be easily deployed from the AWS Serverless application repository. Once the function is deployed, upload logs to your S3 bucket to send them to New Relic.

Requirements

To use our S3 logs shipping Lambda, you only need a New Relic license key

Install the Lambda function

To install the S3 logs shipper Lambda function:

1. Open the AWS Serverless Application Repository in your browser.
2. Make sure that the Lambda is installed in the same region as the S3 bucket.
3. Search for newrelic and check Show apps that create custom IAM roles or resource policies to find NewRelic-log-ingestion-s3.
4. Click the NewRelic-log-ingestion-s3 details and click Deploy.
5. Scroll to the Application settings and enter your New Relic license key. You can also define your logtype, which helps you organize and parse your logs.
6. Confirm that the app creates custom IAM roles and then click Deploy.
7. Once the function is deployed, create a Lambda trigger.

Manual install using Serverless Framework

You can manually install and configure the Lambda from the terminal using the Serverless Framework. If you're not on Linux, install Docker first and keep it running.

# Clone the repository
git clone https://github.com/newrelic/aws_s3_log_ingestion_lambda.git
# Install Serverless and the repo dependencies
npm install -g serverless
npm install
# Set the New Relic License Key env variable
export LICENSE_KEY=YOUR_NEWRELIC_LICENSE_KEY
# Set the logtype (optional)
export LOGTYPE=YOUR_LOGTYPE
# Set the S3 bucket name
export S3_BUCKET_NAME=YOUR_S3_BUCKET_NAME
# The S3 subdirectory (optional)
export S3_PREFIX=YOUR_S3_SUBDIRECTORY
# Deploy the function
serverless deploy

Create a Lambda trigger in AWS

To get your logs streaming to New Relic attach a trigger to the Lambda:

1. In the left side menu, click Functions.
2. Find and select the previously created NewRelic-s3-log-ingestion function.
3. Under Designer, click Add Triggers and select S3 from the dropdown.
4. Select the S3 bucket that contains the log you want to send to New Relic.
5. Optionally configure a prefix and suffix. This is helpful if your logs are in a subdirectory.
6. Check the Enable trigger checkbox, then Add the trigger.

To test the Lambda function, upload a new log file to the S3 bucket you configured in the function settings and then view your log data.

Troubleshooting

Configurations overlap

When deploying the Lambda function, this error may occur:

An error occurred when creating the trigger: Configurations overlap. Configurations on the same bucket cannot share a common event type.

To solve this issue, make sure the bucket doesn't have another event of the same type: Your S3 bucket can't have multiple instances of All object create events.

Cannot connect to the Docker daemon

When using Serverless to deploy the Lambda function from your machine, this error may occur:

Error: docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.

This may mean that either Docker isn't running or that it hasn't been correctly set up. Make sure Docker is running on your machine when deploying the Lambda function using Serverless.

Limitations

• The maximum supported log file size is 35 MB uncompressed.
• The maximum supported individual log line size is 0.25 MB.
• Encrypted log files are not supported.

View log data

If everything is configured correctly and your data is being collected, you should see data logs in both of these places:

• New Relic Logs UI

• New Relic tools for running NRQL queries. For example, you can execute a query like this:

  SELECT * FROM Log

  Copy

What's next?

Now that you've enabled Logs, here are some potential next steps:

• Explore your data using the Logs UI.
• Configure your agent to see contextual log data, such as distributed tracing, stack traces, application logs, and more.
• Query your data and create custom dashboards or alerts.

If no data appears after you enable New Relic Logs, follow the troubleshooting procedures.