AWS VPC Flow Logs monitoring integration

Amazon's Enhanced AWS VPC Flow Logs enables you to capture information about the IP traffic going to and from network interfaces in your VPC. The VPC Flow Logs integration with New Relic allows you to parse all network logs generated by the private networks in order to monitor accepted/rejected traffic in public IPs and inside the VPC itself.

Requirements

Access to this feature depends on your subscription level. Requires Infrastructure Pro.

For the VPC logs to send data to New Relic, you must enable a Lambda function provided by New Relic that will perform the ingestion work. Unlike other AWS integrations that have polling intervals, the VPC Flow Logs integration receives data when it is sent to the Lambda function. The push rate of VPC Flow log data is 15 seconds.

Enable VPC Flow Logs monitoring

In order to send data to the New Relic ingest service, New Relic provides a specific Lambda function that supports pushes from CloudWatch logs, and fetches data from S3 buckets. To assign the Lambda function and enable VPC Flow Logs monitoring:

  1. From the Encryption keys section of the AWS Identity and Access Management (IAM) console, create a new AWS Key Management Services (KMS) encryption key that will be used to encrypt/decrypt your New Relic license key. Optional: Add tags and permissions to the encryption key. Recommendation: Use newrelic-integrations as alias. Use the ID of the new encryption key, which is the last part of the ARN, in step 4.

  2. Create a new AWS Lambda function: Select Lambda > Functions > AWS Serverless Application Repository and use the application called NewRelic-log-ingestion.
  3. Introduce the key ID that was created in step 1 and click Deploy. This action will create a new CloudFormation stack which will create a new function called newrelic-log-ingestion and the required role.
  4. Go to newrelic-log-ingestion function and introduce the LICENSE_KEY environment variable and use your New Relic account license key value. Then, encrypt it as follows:
    • Under Encryption configuration, check the Enable helpers for encryption in transit option and select the newly created newrelic-integrations KMS key.
    • Click the Encrypt button for the LICENSE_KEY environment variable.
  5. (Optional) If you are using New Relic's EU region, add this environment variable to your function: NR_REGION: EU
  6. Stream logs to the Lambda function:

    • From the CloudWatch Management Console, select Logs from the sidebar.
    • Select /aws/vpc/flow-logs and click Actions > Stream to AWS Lambda.
    • Select the New Relic Lambda function you created, in step 3 above (newrelic-log-ingestion) then Next.
    • Keep the default Log format (Amazon VPC Flow Logs) and select Next.
    • Review the configuration and click Start Streaming.

Traffic log configuration

Traffic logs can be configured from within AWS in three modes:

Type Description
Accepted traffic Logs will only capture traffic in the right
Rejected traffic Logs will only reflect rejected traffic
All traffic Logs will show both accepted and rejected traffic

Polling frequency

Unlike other AWS integrations that have polling intervals, the VPC Flow Logs integration receives data when it is sent to the Lambda function. The push rate of VPC Flow log data is 15 seconds.

Amazon VPC Flow Logs data processed

New Relic collects only these log fields from the AWS VPC Flow Log records.

Field Description
version The VPC Flow Logs version.
account-id The AWS account ID for the flow log.
interface-id The ID of the network interface for which the log stream applies.
srcaddr The source IPv4 or IPv6 address. The IPv4 address of the network interface is always its private IPv4 address.
dstaddr The destination IPv4 or IPv6 address. The IPv4 address of the network interface is always its private IPv4 address.
srcport The source port of the traffic.
dstport The destination port of the traffic.
protocol The IANA protocol number of the traffic. For more information, go to Assigned Internet Protocol Numbers.
packets The number of packets transferred during the capture window.
bytes The number of bytes transferred during the capture window.
start The time, in Unix seconds, of the start of the capture window.
end The time, in Unix seconds, of the end of the capture window.
action

The action associated with the traffic:

  • ACCEPT: The recorded traffic was permitted by the security groups or network ACLs.
  • REJECT: The recorded traffic was not permitted by the security groups or network ACLs.
log-status

The logging status of the flow log:

  • OK: Data is logging normally to CloudWatch Logs.
  • NODATA: There was no network traffic to or from the network interface during the capture window.
  • SKIPDATA: Some flow log records were skipped during the capture window. This may be because of an internal capacity constraint, or an internal error.

VPC Flow Log metrics

New Relic processes these traffic metrics:

Metrics Description
provider.bytes The number of bytes.
provider.packets The number of packets.

VPC Flow Log dimensions

New Relic allows you to slice and dice metrics for accepted or rejected traffic using these dimensions:

Dimensions Definition
provider.action If the packet was accepted or rejected
provider.destinationAddress Destination IP address
provider.destinationPort The destination port
provider.interfaceId The network interface ID where the packet is registered
provider.privateDnsName The private DNS name
provider.privateIp The private IP
provider.protocol The internet protocol number
provider.publicDnsName The public DNS name
provider.publicIp The public IP
provider.requesterManaged Indicator that the network interface was created by the user or by AWS
provider.sourceAddress The source IP address
provider.sourcePort The source port
provider.subnetId The subnet ID
provider.vpcId The VPC ID where the network interface belongs

For more help

Recommendations for learning more: