AWS VPC Flow Logs monitoring integration

Amazon's Enhanced AWS VPC Flow Logs enables you to capture information about the IP traffic going to and from network interfaces in your VPC. The VPC Flow Logs integration with New Relic allows you to parse all network logs generated by the private networks in order to monitor accepted/rejected traffic in public IPs and inside the VPC itself.

Requirements

Access to this feature depends on your subscription level. Requires Infrastructure Pro.

For the VPC logs to send data to New Relic, you must enable a Lambda function provided by New Relic that will perform the ingestion work. Unlike other AWS integrations that have polling intervals, the VPC Flow Logs integration receives data when it is sent to the Lambda function. The push rate of VPC Flow log data is 15 seconds.

Enable VPC Flow Logs monitoring

In order to send data to the New Relic ingest service, New Relic provides a specific Lambda function that supports pushes from CloudWatch logsand fetches data from S3 buckets. To assign the Lambda function and enable VPC Flow Logs monitoring:

  1. Create a new AWS Lambda function: Select Lambda > Functions > AWS Serverless Application Repository and use the application called NewRelic-log-ingestion.
  2. (Optional) Introduce your New Relic account license key, which is used to populate the LICENSE_KEY environment variable.
  3. Select Deploy to create a new CloudFormation stack, a new function called newrelic-log-ingestion, and the required role.
  4. Go to newrelic-log-ingestion function. If you haven't already introduced your New Relic account license key, add it now to the LICENSE_KEY environment variable.
  5. (Optional) If you are using New Relic's EU region, add the NR_REGION: EU environment variable to your function.
  6. Continue with the procedure to stream logs to the Lambda function.

Stream logs to Lambda function

To stream logs to the Lambda function:

  1. From the CloudWatch Management Console, select Logs.
  2. Select /aws/vpc/flow-logs and click Actions > Stream to AWS Lambda.
  3. Select the New Relic Lambda function you created (newrelic-log-ingestion) when you enabled VPC Flow Logs monitoring, then select Next.
  4. Keep the default Log format (Amazon VPC Flow Logs) and select Next.
  5. Review the configuration, then select Start streaming.

Configure traffic logs

You can configure traffic logs from within AWS in three modes:

Type Description
Accepted traffic Logs will only capture traffic in the right
Rejected traffic Logs will only reflect rejected traffic
All traffic Logs will show both accepted and rejected traffic

Polling frequency

Unlike other AWS integrations that have polling intervals, the VPC Flow Logs integration receives data when it is sent to the Lambda function. The push rate of VPC Flow log data is 15 seconds.

Amazon VPC Flow Logs data processed

New Relic collects only these log fields from the AWS VPC Flow Log records.

Field Description
version The VPC Flow Logs version.
account-id The AWS account ID for the flow log.
interface-id The ID of the network interface for which the log stream applies.
srcaddr The source IPv4 or IPv6 address. The IPv4 address of the network interface is always its private IPv4 address.
dstaddr The destination IPv4 or IPv6 address. The IPv4 address of the network interface is always its private IPv4 address.
srcport The source port of the traffic.
dstport The destination port of the traffic.
protocol The IANA protocol number of the traffic. For more information, go to Assigned Internet Protocol Numbers.
packets The number of packets transferred during the capture window.
bytes The number of bytes transferred during the capture window.
start The time, in Unix seconds, of the start of the capture window.
end The time, in Unix seconds, of the end of the capture window.
action

The action associated with the traffic:

  • ACCEPT: The recorded traffic was permitted by the security groups or network ACLs.
  • REJECT: The recorded traffic was not permitted by the security groups or network ACLs.
log-status

The logging status of the flow log:

  • OK: Data is logging normally to CloudWatch Logs.
  • NODATA: There was no network traffic to or from the network interface during the capture window.
  • SKIPDATA: Some flow log records were skipped during the capture window. This may be because of an internal capacity constraint, or an internal error.

VPC Flow Log metrics

New Relic processes these traffic metrics:

Metrics Description
provider.bytes The number of bytes.
provider.packets The number of packets.

VPC Flow Log dimensions

New Relic allows you to slice and dice metrics for accepted or rejected traffic using these dimensions:

Dimensions Definition
provider.action If the packet was accepted or rejected
provider.destinationAddress Destination IP address
provider.destinationPort The destination port
provider.interfaceId The network interface ID where the packet is registered
provider.privateDnsName The private DNS name
provider.privateIp The private IP
provider.protocol The internet protocol number
provider.publicDnsName The public DNS name
provider.publicIp The public IP
provider.requesterManaged Indicator that the network interface was created by the user or by AWS
provider.sourceAddress The source IP address
provider.sourcePort The source port
provider.subnetId The subnet ID
provider.vpcId The VPC ID where the network interface belongs

For more help

Recommendations for learning more: