Problem
After installation of the ktranslate network monitoring agent, you're having issues collecting network flow telemetry.
Background
ktranslate returns the raw flow telemetry collected, without editing any of the packet payload. There are several types of flow supported out of the box, with the most prominent being NetFlow v5, NetFlow v9, sFlow, and IPFIX.
All network flow telemetry is stored in the KFlow event type. You can query this directly in NRQL. If this event type is absent, that indicates your account is not receiving the data.
FROM KFlow SELECT *Solution
The ktranslate agent can only collect one type of flow template configured by the -nf.source argument at runtime, which defaults to auto. This tells ktranslate to expect any templates from NetFlow v5 | NetFlow v9 | sFlow | IPFIX so it can translate the packets. A common problem is setting ktranslate to listen to a specific type of flow template, and then exporting another type into the agent. You need to run separate containers for any templates not covered by auto.
Another common misstep here is exporting your flow telemetry to a ktranslate agent, while targeting multiple ports as a destination. In this scenario, you need to run multiple ktranslate agents, with each set to a different value for -nf.port at runtime (default is 9995). You may also need to update the flow exporter configuration on your source network devices to target their specific port.
Each vendor will have their own documentation on properly configuring their devices for the export of network flows. The more advanced versions like NetFlow v9, IPFIX, and sFlow have options that allow administrators to customize the fields being collected and exported. Editing these can effectively disable the ability to correctly process the flow records by ktranslate.
The following fields are required:
- Protocol (Field Type Number:
4) - Layer 4 protocol - Source Address (Field Type Number:
8,27) - Source IPv4 or IPv6 address - Source Port (Field Type Number:
7) - Source TCP/UDP port - Destination Address (Field Type Number:
12,28) - Destination IPv4 or IPv6 address - Destination Port (Field Type Number:
11) - Destination TCP/UDP port
- Interface Receive (Field Type Number:
10) - SNMP index for ingress interface - Interface Transmit (Field Type Number:
14) - SNMP index for egress interface
- Delta Bytes (Field Type Number:
1) - Delta bytes - Total Bytes (Field Type Number:
85) - Total bytes - Out Bytes (Field Type Number:
23) - Out bytes - Initiator Octets (Field Type Number:
231) - Initiator bytes - Responder Octets (Field Type Number:
232) - Responder bytes
- Delta Packets (Field Type Number:
2) - Delta packets - Total Packets (Field Type Number:
86) - Total packets - Out Packets (Field Type Number:
24) - Out packets - Initiator Packets (Field Type Number:
298) - Initiator packets - Responder Packets (Field Type Number:
299) - Responder packets
- ToS (Field Type Number:
5) - Type of service - Source AS (Field Type Number:
16) - Source BGP autonomous system number - Destination AS (Field Type Number:
17) - Destination BGP autonomous system number - Peer Source AS (Field Type Number:
129) - Peer source BGP autonomous system number - Peer Destination AS (Field Type Number:
128) - Peer destination BGP autonomous system number
Tip
ktranslate defaults all flow records to Direction: ingress unless the record explicitly uses a value of egress. This covers various situations where flow records are sent without the Direction field.
Each vendor will have their own documentation on watching flow export counters via their device CLI/UI. A lack of counter growth on the device indicates the flow export is not configured correctly on the device.
Both the Docker container and Linux service deployment options for ktranslate use the host's network to receive data on the mapped port. In order to validate flow records are being received by the host, you can use the tcpdump utility to create a packet capture (.pcap) file that you can later investigate in Wireshark.
Executing this command will set tcpdump to capture every incoming packet across all interfaces on the host, and write the output to a file in the current directory:
$sudo tcpdump -s 0 -i any -w dump_capture.pcapNote that you can add multiple arguments to tcpdump, the most important item here is the output file that you can use for analysis later. Outputting the results in STDOUT creates limited value for the current purposes.
Once you have this file, the next section will show you how to analyze the results.
Tip
One of the most common issues found is a network configuration/firewall rule that blocks the packets from the source network devices to the target ktranslate host. If you're not getting any results with the tcpdump utility, the best place to start troubleshooting is to confirm your network rules and iptables configuration.
Follow these steps to use Wireshark for inspecting the packet capture file.
Start the Wireshark application and open the packet capture file
The initial view shows all of the captured packets, but for flow analysis you'll need to set the application to decode them properly. Using the menu, navigate to Analyze > Decode As..., which opens up a new popup.
On the popup, click the plus (+) icon in the bottom left, which will add a new row to the panel. The initial option in the Current column is (none). Click this to open a drop-down menu and then select either CFLOW for NetFlow and IPFIX, or sFlow for sFlow packets. Click OK in the bottom right to go back to the main UI.
Tip
This menu is alphabetized with case-sensitivity. The sFlow option is at the very bottom of the list.
In the main UI, you should be able to see the CFLOW | sFlow packets now by identifying them in the Protocol column. Applying the display filter (cflow or sflow) will automatically isolate the packets you need from anything else that may be in the capture file.
The following sections outline how to inspect each packet type.
NetFlow and IPFIX protocols use a template approach where the administrator can identify which fields to collect, from a list of standard options. Analysis of these packets is done to ensure the required fields for ktranslate are being captured.
In the main UI for Wireshark, click to select a single CFLOW packet and then expand the section titled FlowSet n, where n is an integer that identifies a singular flow record in a packet. You will then expand the Flow n subgroup to analyze the fields of this flow record.
Tip
You can also click on the Template Frame link in the packet to be taken to a captured packet which contains the template for all flows from this device.
Due to protocol differences between sFlow and the more traditional NetFlow/IPFIX protocols, there are different fields to analyze.
In the main UI for Wireshark, click to select a single sFlow packet and then expand the section titled InMon sFlow. The following fields should be present:
Field | Description |
|---|---|
Datagram version | The version of this sFlow packet. |
Agent address type | IPv4 (1) or IPv6 (2) |
Agent address | IP address the flows are being exported from. This is where you have your flow exporter configured. |
Sub-agent ID | In sFlow v5, you can run multiple exporter processes. This is their unique identifier. |
Sequence number | The number of sFlow packets sent by the agent device. |
SysUptime | Time since the agent device last rebooted. |
NumSamples | The count of sFlow samples contained in the current packet. |
Expanding a sub-group entitled Flow sample will show these additional fields:
Field | Description |
|---|---|
Enterprise | This field annotates custom sFlow enterprise configurations that administrators can optionally enable when configuring their sFlow exports. ( |
sFlow sample type | This is the designation for the type of sample being used when an enterprise customizes their sFlow exports. You can find definitions in the sFlow documentation. |
Sample length | Length of the sample, in bytes. |
Sequence number | Counter value incremented every time the agent takes a sample. |
Sampling rate | 1 in |
Sample pool | Total possible packets that could have been sampled, including the actual sampled packets. |
Dropped packets | Number of packets that were dropped due to resource constraints. |
Input interface | SNMP ifIndex of the interface the packet arrived in from. |
Flow record | Number of sampled records contained in this sample. |
Tip
This same pattern, without the Decode as... step, can be applied to validate receipt of both syslog and SNMP trap data.