Set up network flow data monitoring

You can use our guided install process to install the network flow monitoring agent, or install the agent manually. This doc covers prerequisites to start this install process and a step-by-step walk through of your install options.

Prerequisites

Before you can start, you'll need to sign up for a New Relic account. If you choose to install the agent manually, you also need:

A New Relic account ID.
A New Relic .

Supported types of network flow data

Network flow monitoring supports the four primary types of network flow data and their derivatives. When running the agent, you can specify which major type you want to monitor using the -nf.source option.

Tip

Collection of NetFlow v5, NetFlow v9, sFlow, and IPFIX templates can all be handled using -nf.source.=auto on a single agent. This is enabled as a default setting when using the nr1.flow argument at runtime.

Network flow type	Enabled with `auto`?	`-nf.source` value
AppFlow	✓	`auto` \| `netflow5`
Argus	✓	`auto` \| `netflow5`
Cisco ASA		`asa`
Cisco NBAR		`nbar`
cflowd	✓	`auto` \| `netflow5`
IPFIX	✓	`auto` \| `ipfix`
J-Flow	✓	`auto` \| `netflow5`
NetFlow v5	✓	`auto` \| `netflow5`
NetFlow v9	✓	`auto` \| `netflow9`
NetStream	✓	`auto` \| `netflow5`
Palo Alto Networks		`pan`
RFlow	✓	`auto` \| `netflow5`
sFlow	✓	`auto` \| `sflow`

When should you scale network flow collection?

When planning your strategy for collecting network flows at scale, the following items should be considered:

The ktranslate agent can only perform a single job at a time. An agent running SNMP collection cannot also listen for network flows.
The ktranslate agent can only listen for incoming network flows on a single listening port at a time (default: 9995). If you require multiple ports to be open, each requires a dedicated agent, using the -nf.port configuration option at runtime to change the port.
The default -nf.source=auto configuration allows the container to listen for multiple standard flow types. If you need to parse other types of flow data like Cisco ASA, Cisco NBAR, or Palo Alto Networks templates, each will require their own agent.
New Relic recommends 1 CPU per 2000 flows-per-second (120,000 flows-per-minute). Deciding whether to horizontally scale multiple agents to distribute load or vertically scale a few larger agents to consolidate management is a matter of personal preference.

For most use cases, we recommended our guided install to set up network flow data monitoring. If your set up is more advanced with custom configurations, then we'd recommend installing manually.

Go to one.newrelic.com > All capabilities > Add more data
Scroll down until you see Network and click Network Flows.
Follow the steps outlined in the guided installation process. You can use Docker, Podman, or Linux.
Add network flow data
one.newrelic.com > All capabilities > Add more data > Network > Network Flows to set up network flow data monitoring.
Investigate your network flow data in the New Relic UI.

Before reading about installing the network flow agent manually, consider using our guided install process to avoid errors:

Add network flow data

On a Linux host with Docker installed, download the ktranslate image by running one of the following:
- Docker Hub:
  bash
```
$docker pull kentik/ktranslate:v2
```
- Quay.io:
  bash
```
$docker pull quay.io/kentik/ktranslate:v2
```
Copy the snmp-base.yaml file to the local $HOME directory of your Docker user, and discard the container by running:
bash
```
$cd ~
$id=$(docker create kentik/ktranslate:v2)
$docker cp $id:/etc/ktranslate/snmp-base.yaml .
$docker rm -v $id
```
Edit the snmp-base.yaml file, and add your network flow devices inside the devices dictionary key with the following structure:
```
devices:
  # This key and the corresponding 'device_name'
  # need to be unique for each device
  flow_device1:
    device_name: flow_device1
    device_ip: x.x.x.x/yy
    flow_only: true
    # Optional user tags
    user_tags:
      owning_team: net_eng
      environment: production
```
Important
If you're already monitoring SNMP data devices that will also send network flows, you'll want to ensure that the value for device_name is identical for both configuration files to ensure the flow telemetry is attributed to the right entity in the New Relic UI.

Run ktranslate to listen for network flows with the command:

bash

$docker run -d --name ktranslate-$CONTAINER_SERVICE --restart unless-stopped --pull=always --net=host \
>-v `pwd`/snmp-base.yaml:/snmp-base.yaml \
>-e NEW_RELIC_API_KEY=$YOUR_NR_LICENSE_KEY \
>kentik/ktranslate:v2 \
>  -snmp /snmp-base.yaml \
>  -nr_account_id=$YOUR_NR_ACCOUNT_ID \
>  -metrics=jchf \
>  -tee_logs=true \
>  # Use this field to create a unique value for `tags.container_service` inside of New Relic
>  -service_name=$CONTAINER_SERVICE \
>  -flow_only=true \
>  nr1.flow

Investigate your network flow data in the New Relic UI.

On a host with Podman installed, download the ktranslate image by running the following command:
- Docker Hub:
  bash
```
$podman pull docker.io/kentik/ktranslate:v2
```
Copy the snmp-base.yaml file to the local $HOME directory of your Podman user, and discard the container by running:
bash
```
$cd ~
$id=$(podman create kentik/ktranslate:v2)
$podman cp $id:/etc/ktranslate/snmp-base.yaml .
$podman rm -v $id
```
Edit the snmp-base.yaml file, and add your network flow devices inside the devices dictionary key with the following structure:
```
devices:
  # This key and the corresponding 'device_name'
  # need to be unique for each device
  flow_device1:
    device_name: flow_device1
    device_ip: x.x.x.x/yy
    flow_only: true
    # Optional user tags
    user_tags:
      owning_team: net_eng
      environment: production
```
Important
If you're already monitoring SNMP data devices that will also send network flows, you'll want to ensure that the value for device_name is identical for both configuration files to ensure the flow telemetry is attributed to the right entity in the New Relic UI.

Run ktranslate to listen for network flows with the command:

bash

$podman run -d --name ktranslate-$CONTAINER_SERVICE --userns=keep-id --restart unless-stopped --pull=always --net=host \
>-v `pwd`/snmp-base.yaml:/snmp-base.yaml \
>-e NEW_RELIC_API_KEY=$YOUR_NR_LICENSE_KEY \
>kentik/ktranslate:v2 \
>  -snmp /snmp-base.yaml \
>  -nr_account_id=$YOUR_NR_ACCOUNT_ID \
>  -metrics=jchf \
>  -tee_logs=true \
>  # Use this field to create a unique value for `tags.container_service` inside of New Relic
>  -service_name=$CONTAINER_SERVICE \
>  -flow_only=true \
>  nr1.flow

Tip

Rootless Podman containers aren't able to bind to ports under 1024. If your network flows are sent on a port under 1024 instead of the default (9995), you'll need to create an iptables rule to handle packet redirection with the command:

bash

$sudo iptables -t nat -A PREROUTING -p udp --dport $srcPort -j REDIRECT --to-port 9995

Investigate your network flow data in the New Relic UI.

Depending on your package manager, use one of the commands below to install ktranslate

Yum:

bash

$curl -s https://packagecloud.io/install/repositories/kentik/ktranslate/script.rpm.sh | sudo bash && \
>  sudo yum install ktranslate

Apt:

bash

$curl -s https://packagecloud.io/install/repositories/kentik/ktranslate/script.deb.sh | sudo bash && \
>  sudo apt-get install ktranslate

Define the environment variables used by ktranslate:

bash

$sudo tee "/etc/default/ktranslate.env" > /dev/null <<'EOF'
$NR_ACCOUNT_ID=$YOUR_NR_ACCOUNT_ID
$NEW_RELIC_API_KEY=$YOUR_NR_LICENSE_KEY
$KT_FLAGS="-snmp /etc/ktranslate/snmp-base.yaml \
>  -metrics=jchf \
>  -flow_only=true \
>  -tee_logs=true \
>  -service_name=$CONTAINER_SERVICE \
>  nr1.flow"
$EOF
$
$# ensure /etc/default/ktranslate.env is owned by ktranslate user
$sudo chown ktranslate:ktranslate /etc/default/ktranslate.env

If you don't have an existing snmp-base.yaml configuration file, create one with:
bash
```
$cd ~
$touch snmp-base.yaml
```

Edit the snmp-base.yaml file, and add your network flow devices inside the devices dictionary key with the following structure:

devices:
  # This key and the corresponding 'device_name'
  # need to be unique for each device
  flow_device1:
    device_name: flow_device1
    device_ip: x.x.x.x/yy
    flow_only: true
    # Optional user tags
    user_tags:
      owning_team: net_eng
      environment: production

Restart the ktranslate service to apply your changes to the snmp-base.yaml file:
bash
```
$sudo systemctl restart ktranslate
```
Investigate your network flow data in the New Relic UI.

Find and use your metrics

All network flow logs exported from the ktranslate container use the KFlow namespace, via the New Relic Event API. Currently, these are the default fields populated from this integration:

Attribute	Type	Description
`application`	String	The class of program generating the traffic in this flow record. This is derived from the lowest numeric value from `l4_dst_port` and `l4_src_port`. Common examples include `http`, `ssh`, and `ftp`.
`device_name`	String	The display name of the sampling device for this flow record.
`dst_addr`	String	The target IP address for this flow record.
`dst_as`	Numeric	The target [Autonomous System Number](https://www.iana.org/assignments/ as-numbers/as-numbers.xhtml) for this flow record.
`dst_as_name`	String	The target [Autonomous System Name](https://www.iana.org/assignments/ as-numbers/as-numbers.xhtml) for this flow record.
`dst_endpoint`	String	The target `IP:Port` tuple for this flow record. This is a combination of `dst_addr` and `l4_dst_port`.
`dst_geo`	String	The target country for this flow record, if known.
`in_bytes`	Numeric	The number of bytes transferred for ingress flow records.
`in_pkts`	Numeric	The number of packets transferred for ingress flow records.
`input_port`	Numeric	`If_Index` value for the interface at the source of this flow record.
`l4_dst_port`	Numeric	The target port for this flow record.
`l4_src_port`	Numeric	The source port for this flow record.
`output_port`	Numeric	`If_Index` value for the interface at the destination of this flow record.
`protocol`	String	The display name of the protocol used in this flow record, derived from the [numeric IANA protocol number](https://www.iana.org/assignments/ protocol-numbers/protocol-numbers.xhtml).
`provider`	String	This attribute is used to uniquely identify various sources of data from `ktranslate`. Network flow logs will always have the value of `kentik-flow-device`.
`sample_rate`	Numeric	Sampling rate applied by either the sampling device configuration, or the `sample_rate` argument in `ktranslate`.
`src_addr`	String	The source IP address for this flow record.
`src_as`	Numeric	The source [Autonomous System Number](https://www.iana.org/assignments/ as-numbers/as-numbers.xhtml) for this flow record.
`src_as_name`	String	The source [Autonomous System Name](https://www.iana.org/assignments/ as-numbers/as-numbers.xhtml) for this flow record.
`src_endpoint`	String	The source `IP:Port` tuple for this flow record. It's a combination of `src_addr` and `l4_src_port`.
`src_geo`	String	The source country for this flow record, if known.
`tcp_flags`	Numeric	TCP flags in this flow record.
`timestamp`	Numeric	The time, in Unix seconds, when this flow record was received by the New Relic Event API.

Did this doc help with your installation?

What's next?

You can set up some additional agents to complement your network flow data:

To get better visibility into your network device performance, set up SNMP data monitoring.
To get better visibility into your network syslog messages, set up network syslog monitoring.

Set up network flow data monitoring

Prerequisites

Docker prerequisites

Podman prerequisites

Linux host prerequisites

Network flow data devices prerequisites

Network security prerequisites

Supported types of network flow data

Tip

Network flow types

When should you scale network flow collection?