Amazon VPC Flow Logs monitoring integration

EOL NOTICE

Later this year, we'll be discontinuing support for this integration. You can now set up your AWS VPC flow logs by installing a bundle that includes a dashboard designed for AWS VPC flow logs. See how to set up AWS VPC flow log monitoring.

Amazon's Enhanced AWS VPC Flow Logs enables you to capture information about the IP traffic going to and from network interfaces in your VPC. The VPC Flow Logs integration with New Relic allows you to parse all network logs generated by the private networks in order to monitor accepted/rejected traffic in public IPs and inside the VPC itself.

The New Relic VPC Flow Logs integration can only process logs in AWS's default format. For more information on VPC Flow Logs formatting, see Amazon's VPC Flow Logs documentation.

Requirements

Important

If you integrated an AWS account both through the Metric streams and API Polling mode, you can only see the VPC logs in the provider account using the Metric streams integration.

For the VPC logs to send data to New Relic, you must enable a Lambda function provided by New Relic that will perform the ingestion work. Unlike other AWS integrations that have polling intervals, the VPC Flow Logs integration receives data when it is sent to the Lambda function. The push rate of VPC Flow log data is 15 seconds.

Enable VPC Flow Logs monitoring

In order to send data to the New Relic ingest service, New Relic provides a specific Lambda function that supports pushes from CloudWatch logs and fetches data from S3 buckets. To assign the Lambda function and enable VPC Flow Logs monitoring:

Create a new AWS Lambda function from the Serverless Repository: Go to Lambda > Create Function > Browse serverless App repository, check the box for Show apps that create custom IAM roles or resource policies, and search for NewRelic-log-ingestion.
Populate the LICENSE_KEY environment variable with your New Relic account .
Review all optional parameters and adapt them based on your use cases.
Select Deploy to create a new CloudFormation stack, a new function called newrelic-log-ingestion, and the required role.
Go to the newrelic-log-ingestion function.
Continue with the procedure to stream logs to the Lambda function.

Tip

The newrelic-log-ingestion function requires the AWSLambdaBasicExecutionRole policy which contains the minimum permissions (as recommended by AWS). A custom IAM role name can be defined at install time, otherwise, an appropriate Role will be created, which will require CAPABILITY_IAM to be acknowledged.

Stream logs to Lambda function

To stream logs to the Lambda function:

From the CloudWatch Management Console, select Logs.
Select /aws/vpc/flow-logs and click Actions > Stream to AWS Lambda.
Select the New Relic Lambda function you created (newrelic-log-ingestion) when you enabled VPC Flow Logs monitoring, then select Next.
Keep the default Log format (Amazon VPC Flow Logs) and select Next.
Review the configuration, then select Start streaming.

Configure traffic logs

You can configure traffic logs from within AWS in three modes:

Type	Description
Accepted traffic	Logs will only capture traffic in the right
Rejected traffic	Logs will only reflect rejected traffic
All traffic	Logs will show both accepted and rejected traffic

Polling frequency

Unlike other AWS integrations that have polling intervals, the VPC Flow Logs integration receives data when it is sent to the Lambda function. The push rate of VPC Flow log data is 15 seconds.

Amazon VPC Flow Logs data processed

New Relic collects only these log fields from the Amazon VPC Flow Log records.

Field	Description
`version`	The VPC Flow Logs version.
`account-id`	The AWS account ID for the flow log.
`interface-id`	The ID of the network interface for which the log stream applies.
`srcaddr`	The source IPv4 or IPv6 address. The IPv4 address of the network interface is always its private IPv4 address.
`dstaddr`	The destination IPv4 or IPv6 address. The IPv4 address of the network interface is always its private IPv4 address.
`srcport`	The source port of the traffic.
`dstport`	The destination port of the traffic.
`protocol`	The IANA protocol number of the traffic. For more information, go to Assigned Internet Protocol Numbers.
`packets`	The number of packets transferred during the capture window.
`bytes`	The number of bytes transferred during the capture window.
`start`	The time, in Unix seconds, of the start of the capture window.
`end`	The time, in Unix seconds, of the end of the capture window.
`action`	The action associated with the traffic: `ACCEPT`: The recorded traffic was permitted by the security groups or network ACLs. `REJECT`: The recorded traffic was not permitted by the security groups or network ACLs.
`log-status`	The logging status of the flow log: OK: Data is logging normally to CloudWatch Logs. `NODATA`: There was no network traffic to or from the network interface during the capture window. `SKIPDATA`: Some flow log records were skipped during the capture window. This may be because of an internal capacity constraint, or an internal error.

VPC Flow Log metrics

New Relic processes these traffic metrics:

Metrics	Description
`provider.bytes`	The number of bytes.
`provider.packets`	The number of packets.

VPC Flow Log dimensions

New Relic allows you to slice and dice metrics for accepted or rejected traffic using these dimensions:

Dimensions	Definition
`provider.action`	If the packet was accepted or rejected
`provider.destinationAddress`	Destination IP address
`provider.destinationPort`	The destination port
`provider.interfaceId`	The network interface ID where the packet is registered
`provider.privateDnsName`	The private DNS name
`provider.privateIp`	The private IP
`provider.protocol`	The internet protocol number
`provider.publicDnsName`	The public DNS name
`provider.publicIp`	The public IP
`provider.requesterManaged`	Indicator that the network interface was created by the user or by AWS
`provider.sourceAddress`	The source IP address
`provider.sourcePort`	The source port
`provider.subnetId`	The subnet ID
`provider.vpcId`	The VPC ID where the network interface belongs