Fluentd plugin for log forwarding

If your log data is already being monitored by Fluentd, you can use our Fluentd integration to forward and enrich your log data in New Relic.

Forwarding your Fluentd logs to New Relic will give you enhanced log management capabilities to collect, process, explore, query, and alert on your log data.

Basic process

To enable log management capabilities:

1. Make sure you have:

   • A New Relic license key
   • Fluentd 1.0 or higher

2. Install the Fluentd plugin.

3. Configure the Fluentd plugin.

4. If applicable, configure EU endpoint

5. Test the Fluentd plugin.

6. Generate some traffic and wait a few minutes, then check your account for data.

Install the Fluentd plugin

To install the Fluentd plugin:

$
fluent-gem install fluent-plugin-newrelic

$
td-agent-gem install fluent-plugin-newrelic

Configure the Fluentd plugin

To configure your Fluentd plugin:

1. In your fluent.conf file, or td-agent.conf if using the td-agent, add the following block of data, replacing the placeholder text with your license key:

   #Tail one or more log files
   <source>
     @type tail
     <parse>
       @type none
     </parse>
     path /path/to/file
     tag example.service
   </source>
   
   #Add hostname and service_name to all events with "example.service" tag
   <filter example.service>
     @type record_transformer
     <record>
       service_name ${tag}
       hostname "#{Socket.gethostname}"
     </record>
   </filter>
   
   #Forward all events to New Relic
   
   <match>
     @type newrelic
     license_key YOUR_LICENSE_KEY
   </match>

   Copy

2. Restart the Fluentd service to ensure your changes are applied.

Configure the Fluentd Plugin for EU accounts

By default the Fluentd plugin forwards logs to our US endpoint: https://log-api.newrelic.com/log/v1. If your New Relic organization is in our EU data center, you must manually set the base_uri property to the EU endpoint. For example:

#Tail one or more log files
<source>
  @type tail
  <parse>
    @type none
  </parse>
  path /path/to/file
  tag example.service
</source>

#Add hostname and service_name to all events with "example.service" tag
<filter example.service>
  @type record_transformer
  <record>
    service_name ${tag}
    hostname "#{Socket.gethostname}"
  </record>
</filter>

#Forward all events to New Relic EU Endpoint
<match>
  @type newrelic
  license_key YOUR_LICENSE_KEY
  base_uri https://log-api.eu.newrelic.com/log/v1
</match>

Test the Fluentd plugin

To test if your Fluentd plugin is receiving input from a log file:

1. Run the following command to append a test log message to your log file:

   bash

   Copy

   $
   echo "test message" >> /PATH/TO/YOUR/LOG/FILE

2. Search our logs UI for test message.

View log data

If everything is configured correctly and your data is being collected, you should see logs in both of these places:

• Our logs UI
• Our tools for running NRQL queries. For example, you can execute a query like this:

SELECT * FROM Log

If no data appears after you enable our log management capabilities, follow our standard log troubleshooting procedures.

Tune up log Fluentd buffer

By default, the plugin sends logs to New Relic every five seconds. If you want to change this timing, add a <buffer> block to the configuration by following this example:

[...]

# Forward all events to New Relic EU Endpoint
<match>
  @type newrelic
  license_key YOUR_LICENSE_KEY

  <buffer time>
    timekey 60s
  </buffer>
</match>

For more information, see the Fluentd documentation about buffer configurations.

Configure UTF-16LE to UTF-8 transformation

In this example for Microsoft SQL Server error logs, use Fluentd to send UTF-16LE encoded logs to New Relic with the required UTF-8 encoding for ingest. You can also adopt this example to other encodings.

#Tail one or more log files
<source>
  @type tail
  <parse>
    @type none
  </parse>
  path "D:/sqlserver/error.log"
  tag example.service
  encoding UTF-8
  from_encoding UTF-16LE
</source>

#Add hostname and service_name to all events with "example.service" tag
<filter example.service>
  @type record_transformer
  <record>
    service_name ${tag}
    hostname "#{Socket.gethostname}"
    logtype MSSQL
  </record>
</filter>

#Forward all events to New Relic
<match>
  @type newrelic
  license_key YOUR_LICENSE_KEY
</match>

Configure Shift-JIS to UTF-8 transformation

Japanese log files encoded in Shift-JIS (CP932) are converted to the required UTF-8 encoding and are required for importing and sending log data to New Relic.

<source>
#Tail one or more log files
  @type tail
  path C:¥opt¥fluent¥cp932text.log  # Full path of the log file you want to collect
  tag example.service
  from_encoding CP932 #logFile character encoding
  encoding UTF-8 #Character encoding when sending to New Relic
  <parse>
    @type none
  </parse>
</source>

#Forward all events to New Relic
<match>
  @type newrelic
  license_key YOUR_LICENSE_KEY
</match>

What's next?

Explore logging data across your platform with our logs UI.

• Get deeper visibility into both your application and your platform performance data by forwarding your logs with our logs in context capabilities.
• Set up alerts.
• Query your data and create dashboards.

Disable log forwarding

To disable log forwarding capabilities, follow standard procedures in Fluentd documentation. You do not need to do anything else in New Relic.