Amazon's Enhanced AWS VPC Flow Logs enables you to capture information about the IP traffic going to and from network interfaces in your VPC. The VPC Flow Logs integration with New Relic allows you to parse all network logs generated by the private networks in order to monitor accepted/rejected traffic in public IPs and inside the VPC itself.
The New Relic VPC Flow Logs integration can only process logs in AWS's default format. For more information on VPC Flow Logs formatting, see Amazon's VPC Flow Logs documentation.
If you integrated an AWS account both through the Metric streams and API Polling mode, you can only see the VPC logs in the provider account using the Metric streams integration.
For the VPC logs to send data to New Relic, you must enable a Lambda function provided by New Relic that will perform the ingestion work. Unlike other AWS integrations that have polling intervals, the VPC Flow Logs integration receives data when it is sent to the Lambda function. The push rate of VPC Flow log data is 15 seconds.
In order to send data to the New Relic ingest service, New Relic provides a specific Lambda function that supports pushes from CloudWatch logs and fetches data from S3 buckets. To assign the Lambda function and enable VPC Flow Logs monitoring:
- Create a new AWS Lambda function from the Serverless Repository: Go to Lambda > Create Function > Browse serverless App repository, check the box for Show apps that create custom IAM roles or resource policies, and search for
- Populate the
LICENSE_KEYenvironment variable with your New Relic account license key.
- Review all optional parameters and adapt them based on your use cases.
- Select Deploy to create a new CloudFormation stack, a new function called
newrelic-log-ingestion, and the required role.
- Go to the
- Continue with the procedure to stream logs to the Lambda function.
newrelic-log-ingestion function requires the AWSLambdaBasicExecutionRole policy which contains the minimum permissions (as recommended by AWS).
A custom IAM role name can be defined at install time, otherwise, an appropriate Role will be created, which will require CAPABILITY_IAM to be acknowledged.
To stream logs to the Lambda function:
- From the CloudWatch Management Console, select Logs.
- Select /aws/vpc/flow-logs and click Actions > Stream to AWS Lambda.
- Select the New Relic Lambda function you created (
newrelic-log-ingestion) when you enabled VPC Flow Logs monitoring, then select Next.
- Keep the default Log format (Amazon VPC Flow Logs) and select Next.
- Review the configuration, then select Start streaming.
You can configure traffic logs from within AWS in three modes:
Logs will only capture traffic in the right
Logs will only reflect rejected traffic
Logs will show both accepted and rejected traffic
Unlike other AWS integrations that have polling intervals, the VPC Flow Logs integration receives data when it is sent to the Lambda function. The push rate of VPC Flow log data is 15 seconds.
New Relic collects only these log fields from the Amazon VPC Flow Log records.
The VPC Flow Logs version.
The AWS account ID for the flow log.
The ID of the network interface for which the log stream applies.
The source IPv4 or IPv6 address. The IPv4 address of the network interface is always its private IPv4 address.
The destination IPv4 or IPv6 address. The IPv4 address of the network interface is always its private IPv4 address.
The source port of the traffic.
The destination port of the traffic.
The IANA protocol number of the traffic. For more information, go to Assigned Internet Protocol Numbers.
The number of packets transferred during the capture window.
The number of bytes transferred during the capture window.
The time, in Unix seconds, of the start of the capture window.
The time, in Unix seconds, of the end of the capture window.
The action associated with the traffic:
The logging status of the flow log:
New Relic processes these traffic metrics:
The number of bytes.
The number of packets.
New Relic allows you to slice and dice metrics for accepted or rejected traffic using these dimensions:
If the packet was accepted or rejected
Destination IP address
The destination port
The network interface ID where the packet is registered
The private DNS name
The private IP
The internet protocol number
The public DNS name
The public IP
Indicator that the network interface was created by the user or by AWS
The source IP address
The source port
The subnet ID
The VPC ID where the network interface belongs