Set up Amazon VPC Flow Logs monitoring

Amazon Virtual Private Cloud Flow (VPC) Logs via Amazon Kinesis Data Firehose help you reduce the friction of sending logs to New Relic. With VPC flow logs from across your AWS estates, you can quickly understand key insights for performance analytics and troubleshooting network connectivity.

Amazon Virtual Private Cloud (VPC) enables you to launch AWS resources into an isolated and secure virtual network with the benefits of using scalable AWS infrastructure.

Prerequisites

New Relic prerequisites

• A New Relic account. Don't have one? Sign up for free! No credit card required.

AWS prerequisites

• Environment:
  • AWS CLI (v 1.9.15+) installed.
  • A list of AWS regions where you collect Amazon VPC Flow Logs.
  • VPC IDs that will be configured to ship VPC flow logs. If you want more granular control, specify the subnet IDs instead of the VPC IDs.
• Permissions:
  • An AWS user with permissions to create VPC flow logs.
  • Amazon S3 log file permissions to read and write for the log delivery owner.
  • Allow Kinesis Data Firehose to Assume an IAM Role.
• Amazon Resource Names (ARNs):
  • ARN of an S3 bucket with permissions to store undelivered flow log messages.
  • If you wish to enable sampling, you'll need the ARN for the Lambda's IAM role.
  • ARN for the Kinesis Data Firehose with S3 bucket access.

    Tip

    We recommend you create a new data firehose with our guided install the first time you configure VPC flow logs for a specific region. For subsequent VPCs and subnets in the same region, you can reuse the existing data firehose.

IAM roles

If you set up the flow logs integration using the AWS CLI, you must provide one or two IAM roles for different infrastructure components. If you use CloudFormation, you can either provide your own roles or let the template define new roles.

The necessary roles should have at least the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::<FIREHOSE_ERRORS_BUCKET>",
        "arn:aws:s3:::<FIREHOSE_ERRORS_BUCKET>/*"
      ]
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": ["lambda:InvokeFunction", "lambda:GetFunctionConfiguration"],
      "Resource": [
        "arn:aws:lambda:us-west-2:<YOUR_ACCOUNT>:function:NewRelic-Flow-Sampling-Lambda",
        "arn:aws:lambda:us-west-2:<YOUR_ACCOUNT>:function:NewRelic-Flow-Sampling-Lambda:*"
      ]
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "logs:CreateLogGroup",
      "Resource": "arn:aws:logs:us-west-2:<YOUR_ACCOUNT>:*",
      "Effect": "Allow"
    },
    {
      "Action": ["logs:CreateLogStream", "logs:PutLogEvents"],
      "Resource": [
        "arn:aws:logs:us-west-2:<YOUR_ACCOUNT>:log-group:/aws/lambda/NewRelic-Flow-Sampling-Lambda:*"
      ],
      "Effect": "Allow"
    }
  ]
}

Formatting your logs in New Relic

To use the curated flow log exploration and entity linking, you must follow this format for the VPC flow logs:

$
${version} ${account-id} ${region} ${az-id} ${sublocation-type} ${vpc-id} ${subnet-id} ${instance-id} ${interface-id} ${srcaddr} ${pkt-srcaddr} ${pkt-src-aws-service} ${dstaddr} ${pkt-dstaddr} ${pkt-dst-aws-service} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${flow-direction} ${traffic-path} ${start} ${end} ${action} ${log-status}

You must use a log partition for VPC flow logs named Log_VPC_Flows_AWS. If you use the guided install, this is handled automatically.

Set up Amazon VPC Flow Logs monitoring in New Relic

Follow the guided wizard to install Amazon VPC Flow Logs:

1. Start the guided install process.
2. From the Select an account dropdown, choose the New Relic account you want to send Amazon VPC Flow Logs to, and click Continue.
3. In the Select your data section, choose Amazon VPC Flow Logs and click Continue.
4. In the Select your install method section, continue with CLI and click Continue.

After these steps, a new wizard pops-up to help you set up the sending of Amazon VPC Flow Logs to New Relic through the AWS Kinesis Firehose service.

1. In the Choose Setup Options section:

   • Verify your setup method is correct.
   • Select the AWS region that will send VPC flow logs to New Relic.
   • Optionally, if you are reusing a Kinesis Data Firehose, select the I already have a Kinesis Firehose to New Relic checkbox and continue to the Define flow logs section.
   • Click Continue.

2. In the Define Kinesis Firehose section:

   • In the Kinesis Firehose Name field, make sure the generated name is correct.
   • In the Firehose Backup Bucket field, enter the ARN of the S3 bucket to be used to store messages that fail delivery. The ARN must follow this format: arn:my_string.
   • In the Firehose IAM Role field, enter the ARN of the IAM role to be used by Kinesis Data Firehose. The ARN must follow this format: arn:my_string.
   • Click Continue.
   • Optionally, if you want to sample VPC flow logs, select the Use Sampling checkbox and:

   Use Sampling Lambda

   • First, in the Define Sampling Lambda section, do the following:

     • In the Lambda Name field, make sure the generated name for the Lambda function is correct. If you have specific Lambda naming standards, edit the name.
     • In the Sample Rate field, specify the sample rate to be applied. By default it's set to 1:10.

   • Then, in the Create Sampling Lambda section, click Generate CLI Command and:

     • Copy the contents of the code block for Create your sampling Lambda and run it in the AWS CLI.
     • To check if your Lambda is created, run the command from the second step in the AWS CLI.
     • Copy the returned ARN for the Lambda function and paste into the Sampling Lambda ARN field in the format arn:my_string. Then, click Continue.

     Tip

     Sampling is a simple count function but it'll reduce the volume of flow logs shipped to New Relic. Data sampled by using this Lambda function, may miss a specific conversation in New Relic. Watch out for callouts indicating sampled data in the network monitoring UI of New Relic.

3. In the Generate Kinesis Firehose section, click Generate CLI Command and:

   Tip

   We automatically generate a new license key to be used for this data ingest. To regenerate a key, click Generate and use a new key.

   If you want to re-use an existing key, update the AccessKey value in the first step.

   • Copy the contents of the code block for Create your Kinesis Data Firehose and paste run it in the AWS CLI.
   • To check if your Kinesis Firehose is created, run the command from the second step in the AWS CLI. If no ARN is returned, wait 30 seconds and try again.
   • Copy the returned ARN for the Kinesis Firehose and paste it into the Kinesis Data Firehose ARN field in the format arn:my_string. Then, click Continue.
   

4. In the Define Flow Logs section, do the following:

   • From the Traffic Type dropdown, select whether to send only accepted, only rejected, or all flow log entries.
   • In the Flow Source ID field, enter the VPC ID (vpc-MY_STRING) or the subnet ID (subnet-MY_STRING) for which Amazon VPC Flow Logs should be created.
   • The Flow Source Type field will be automatically populated, so click Continue.

5. In the Create Flow Logs section, click Generate CLI Command and copy the contents of the syntax block. Then, run it in the AWS CLI to begin generating flow logs for the specified resources.

6. Click Continue to start exploring Amazon VPC Flow Logs in the network monitoring section of New Relic.

7. Visualize your network performance data in New Relic.