Fluent Bit plugin for log forwarding

If your log data is already being monitored by Fluent Bit, you can use our Fluent Bit output plugin to forward and enrich your log data in New Relic.

Forwarding your Fluent Bit logs to New Relic will give you enhanced log management capabilities to collect, process, explore, query, and alert on your log data. You have these options to install the Fluent Bit:

• Kubernetes installation
• Using a Docker image
• On-host installation

Kubernetes installation

New Relic has a Fluent Bit output plugin to forward your logs to New Relic log management. You can install this plugin as a standalone Docker image in a Kubernetes cluster, where it functions as a DaemonSet, also known as the Kubernetes plugin.

You can install it in your cluster using our Helm chart in two ways:

• Through our guided install
• Manual installation

Using our guided install

Although the newrelic-logging chart works as a standalone, we recommend installing it as part of the nri-bundle chart.

The best way to install this is through our guided installation process. This guided install generates the Helm commands required to install it.

Manual installation

You can install it manually using Helm, although this option is not recommended. Run this command to install the repo:

$
helm repo add newrelic https://helm-charts.newrelic.com

To update the repo, run this command:

$
helm repo update newrelic

See Uninstall Kubernetes integration if you want to uninstall it.

Using a Docker image

If you've your own custom kubernetes integration, we recommend using our Docker image that comes with the newrelic-fluent-bit-output plugin. Or you can use the Docker image as a base image and layer your own custom configuration files.

On-host installation

For an on-host installation of the Fluent Bit plugin, follow these steps:

1. Open the New Relic's Fluent Bit plugin repository on GitHub.

2. From the repository page, clone or download the repository.

3. Run this command to build your plugin:

   bash

   Copy

   $
   cd newrelic-fluent-bit-output && make all

4. Store out_newrelic.so or out_newrelic_winXX.dll at a location where fluent-bit daemon can access them.

Upgrade the Fluent Bit plugin

Before you upgrade your Fluent Bit plugin, run the following NRQL query to find the current versions of the output plugin you're using in your system:

FROM K8sContainerSample 
SELECT latest(containerImage) 
WHERE podName like '%newrelic-logging%' 
FACET clusterName

To update, follow the installation instructions or grab the latest pre-compiled version from our GitHub repository.

Configure the Fluent Bit plugin

Fluent Bit needs to know the location of the New Relic plugin and the New Relic license key to output data to New Relic.

To configure your Fluent Bit plugin, follow these steps:

1. Locate or create the plugins.conf file in your plugins directory.

2. In the plugins.conf file, add a reference to out_newrelic.so, adjacent to your fluent-bit.conf file:

   [PLUGINS]
       Path /PATH/TO/newrelic-fluent-bit-output/out_newrelic.so

   Copy

3. In the fluent-bit.conf file, add the following line under the service block:

   [SERVICE]
       # This is the main configuration block for fluent bit.
       # Ensure the follow line exists somewhere in the SERVICE block
       Plugins_File plugins.conf

   Copy

4. At the bottom of the fluent-bit.conf file, add the following to set up the input, filter, and output sections. Replace the placeholder text with your license key:

   [INPUT]
       Name tail
       Tag  my.tag
       Path /PATH/TO/YOUR/LOG/FILE
       
   # If you have multiple sources, just add another [INPUT] section like this:
   [INPUT] 
       Name tail 
       Tag  my.other.tag 
       Path /PATH/TO/SOME/OTHER/LOG/FILE 
   
   # Having multiple [FILTER] blocks allows you to control the flow of changes as they read top down. 
   [FILTER]
       Name modify
       # Here we only match on one tag, my.tag, defined in the [INPUT] section earlier
       Match  my.tag
       # Below, we're renaming the host.cpu attribute to CPU
       Rename host.cpu CPU
   
   [FILTER]
       Name   record_modifier
       # Match on all tags, *, so all logs get decorated per the Record clauses below. Record adds attributes + their values to each record.
       Match  *
       # Adding a logtype attribute ensures your logs will be automatically parsed by our built-in parsing rules
       Record logtype nginx
       # Add the server's hostname to all logs generated
       Record hostname ${HOSTNAME}
   
   [OUTPUT]
       Name       newrelic
       Match      *
       licenseKey YOUR_LICENSE_KEY

   Copy

5. Restart your Fluent Bit instance. Run this command:

   bash

   Copy

   $
   fluent-bit -c /PATH/TO/fluent-bit.conf

Test the Fluent Bit plugin

To test if your Fluent Bit plugin is receiving input from a log file:

1. Run this command to append a test log message to your log file:

   bash

   Copy

   $
   echo "test message" >> /PATH/TO/YOUR/LOG/FILE

2. Search for test message in our logs UI .

For more options, see the modify filter on the Fluent Bit documentation. See also our documentation to forward your logs using the infrastructure agent.

Optional: Configure plugin attributes

Once you've installed and configured the Fluent Bit plugin, you can use the following attributes to configure how the plugin sends data to New Relic:

View log data

If you configured everything correctly and New Relic collects your data, you should see log data in both of these places:

• Our logs UI

• Our tools for running NRQL queries. For example, you can run a query like this:

SELECT * 
FROM Log

If you don't get any data after you enable our log management capabilities, follow our standard log troubleshooting procedures.

What's next?

Explore logging data across your platform with our logs UI.

• Get deeper visibility into both your application and your platform performance data by forwarding your logs with our logs in context capabilities.

• Set up alerts.

• Query your data and create dashboards.

Disable log forwarding

To disable log forwarding capabilities, follow standard procedures in Fluent Bit documentation. You don't need to do anything else in New Relic.