Use TCP endpoint to forward logs to New Relic

We offer a wide range of solutions to get your log data into New Relic. But in other situations where you don't have log forwarders (such as CDNs, hardware devices, or managed services), you can use syslog protocols via a TCP endpoint. You can forward your logs to New Relic using syslog clients such as rsyslog and syslog-ng.

Compatibility and requirements

To forward logs to New Relic using a syslog client, you need:

• A valid New Relic license key for the account you want to send logs to
• Some minor changes to the syslog client's configuration, as explained in this document

Limitations

Log lines that exceed 1 MB in compressed size will be dropped. Other limitations from the Log API may also apply. To prevent data loss, ensure your log lines are within this size limit.

Configure rsyslog versions 8 and higher (Amazon Linux, Redhat, Centos)

To forward logs to New Relic with rsyslog for Amazon Linux, Redhat, and Centos distributions:

1. Install packages required to allow rsyslog to send logs using TLS encryption:

   bash

   Copy

   $
   sudo yum install rsyslog-gnutls ca-certificates

2. Optional: Configure rsyslog to tail log files and forward their contents to New Relic. Add the following to the /etc/rsyslog.conf file in order to enable its text file input module:

   module(load="imfile"
          PollingInterval="10"
   )

   Copy

3. In the /etc/rsyslog.d/ directory, create a text file named newrelic.conf.

4. Check whether the $IncludeConfig options under /etc/rsyslog.conf already have a matching wildcard that will include the newly created newrelic.conf file under the /etc/rsyslog.d directory. Otherwise you'll need to explicitly include /etc/rsyslog.d/newrelic.conf in /etc/rsyslog.conf using $IncludeConfig /etc/rsyslog.d/newrelic.conf

5. Add the following to newrelic.conf. Replace the highlighted values. For YOUR_LICENSE_KEY, use your New Relic license key:

   ## Specify each of the files to be tailed in case step 1 is done
   
   input(type="imfile" Tag="YOUR_FILE_TAG" File="PATH_TO_FILE")
   
   ## Template expected by the New Relic Syslog endpoint
   
   template(name="newrelic-rfc5424"
            type="string"
            string="YOUR_LICENSE_KEY <%pri%>%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% %structured-data% %msg%\n"
   )
   
   ## Configure TLS and log forwarding
   
   global(DefaultNetstreamDriver="gtls"
          ##Specify the full path to the system's CA certificate:
          DefaultNetstreamDriverCAFile="<path to certificate>"
          ## RHEL/CentOS/Amazon Linux:
          ##/etc/pki/tls/certs/ca-bundle.crt
          ## Ubuntu:
          ##/etc/ssl/certs/ca-certificates.crt
          ##Other systems:
          ##Specify the full path to the system's generic CA certificate.
   )
   
   action(type="omfwd"
          Target="newrelic.syslog.nr-data.net"
          Port="6514"
          Protocol="tcp"
          Template="newrelic-rfc5424"
          ResendLastMSGOnReconnect="on"
          StreamDriver="gtls"
          StreamDriverAuthMode="x509/name"
          StreamDriverPermittedPeers="*.syslog.nr-data.net"
          StreamDriverMode="1"
   )

   Copy

6. (Optional): Add custom attributes to your syslog data.

   To make your syslog data consistent with the rest of your telemetry data, you can add custom attributes. There are two ways to do this:

   • Add custom attributes directly in the template string. You'll need to escape the double quotes (") for the string values. For example:

     [<STRUCTURED_DATA_ID> <ATTR_NAME_1>="<ATTR_VALUE_1>" <ATTR_NAME_2>="<ATTR_VALUE_2>" ...]

     template(name="newrelic-rfc5424"
              type="string"
              string="YOUR_LICENSE_KEY <%pri%>%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% [someId environment=\"prod\" region=\"US\"] %msg%\n"
     )

     Copy

   • Add custom attributes using a custom property variable in the template string. This method avoids the need to escape string values.

     template(name="newrelic-rfc5424"
              type="string"
              string="YOUR_LICENSE_KEY <%pri%>%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% %!custom-structured-data% %msg%\n"
     )
     
     set $!custom-structured-data = '[someId1 environment="prod" region="US"][someId2 team="sre" cost-center="123"]';

     Copy

Make sure you replace the %structured-data% section in the template string with custom tags/attributes.

7. Restart the rsyslog service by running:

   bash

   Copy

   $
   sudo systemctl restart rsyslog

8. Check your New Relic account for logs.

Configure rsyslog versions 8 and above with compression (Amazon Linux, Redhat, Centos)

To configure rsyslog versions 8 and above for Amazon Linux, Redhat, and Centos distributions:

1. Follow steps 1 to 4 described in configure rsyslog.

   Important

   Note the configured port number is 6516 instead of 6514.

2. Add the following to newrelic.conf. Replace all highlighted values. For YOUR_LICENSE_KEY, use your New Relic license key:

   ## Specify each of the files to be tailed in case step 1 is done
   
   input(type="imfile" Tag="YOUR_FILE_TAG" File="PATH_TO_FILE"
   
   ## Template expected by the New Relic Syslog endpoint
   
   template(name="newrelic-rfc5424"
            type="string"
            string="YOUR_LICENSE_KEY <%pri%>%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% %structured-data% %msg%\n"
   )
   
   ## Configure TLS and log forwarding
   
   global(DefaultNetstreamDriver="gtls"
          ##Specify the full path to the system's CA certificate:
          DefaultNetstreamDriverCAFile="<path to certificate>"
          ## RHEL/CentOS/Amazon Linux:
          ##/etc/pki/tls/certs/ca-bundle.crt
          ## Ubuntu:
          ##/etc/ssl/certs/ca-certificates.crt
          ##Other systems:
          ##Specify the full path to the system's generic CA certificate.
   )
   
   action(type="omfwd"
          Target="newrelic.syslog.nr-data.net"
          Port="6516"
          Protocol="tcp"
          Template="newrelic-rfc5424"
          ResendLastMSGOnReconnect="on"
          StreamDriver="gtls"
          StreamDriverAuthMode="x509/name"
          StreamDriverPermittedPeers="*.syslog.nr-data.net"
          StreamDriverMode="1"
          compression.Mode="stream:always"
   )

   Copy

3. Restart the rsyslog service by running:

   bash

   Copy

   $
   sudo systemctl restart rsyslog

4. Check your New Relic account for logs You can also find logs by searching for the attribute plugin.type:"syslog-newrelic".

Legacy config file for rsyslog versions 7 or lower (Amazon Linux, Redhat, Centos)

If you're using rsyslog version 7 or below, the configuration files need to be adapted to the obsolete legacy format. This format is only kept for retro compatibility purposes. We strongly recommend to avoid using it, as stated in the rsyslog documentation. To configure logs for Amazon Linux, Redhat, and Centos distributions:

1. Install packages required to allow rsyslog to send logs using TLS encryption:

   bash

   Copy

   $
   sudo yum install rsyslog-gnutls ca-certificates

2. Optional: Configure rsyslog to tail log files and forward their contents to New Relic. Add the following to the /etc/rsyslog.conf file in order to enable its text file input module:

   $ModLoad imfile
   $InputFilePollInterval 10
   $PrivDropToGroup adm
   $WorkDirectory /var/spool/rsyslog

   Copy

3. In the /etc/rsyslog.d/ directory, create a text file named newrelic.conf.

4. Check whether the $IncludeConfig options under /etc/rsyslog.conf already have a matching wildcard that will include the newly created newrelic.conf file under the /etc/rsyslog.d directory. Otherwise you'll need to explicitly include /etc/rsyslog.d/newrelic.conf in /etc/rsyslog.conf using $IncludeConfig /etc/rsyslog.d/newrelic.conf.

5. Add the following to newrelic.conf. Replace the highlighted values. For YOUR_LICENSE_KEY, use your New Relic license key:

   ## Template expected by the New Relic Syslog endpoint
   
   $template NRLogFormat,"YOUR_LICENSE_KEY <%pri%>%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid%  %structured-data% %msg%\n"
   
   ## Specify each of the files to be tailed in case step 1 is done
   
   $InputFileName /path/to/file
   $InputFileTag YOUR_FILE_TAG
   $InputFileSeverity info
   $InputRunFileMonitor
   
   ## Configure TLS and log forwarding
   ##Specify the full path to the system's CA certificate:
   $DefaultNetstreamDriverCAFile <path to certificate>
   ## RHEL/CentOS/Amazon Linux:
   ##/etc/pki/tls/certs/ca-bundle.crt
   ## Ubuntu:
   ##/etc/ssl/certs/ca-certificates.crt
   ##Other systems:
   ##Specify the full path to the system's generic CA certificate.
   $ActionSendStreamDriver gtls
   $ActionSendStreamDriverMode 1
   $ActionSendStreamDriverAuthMode x509/name
   $ActionSendStreamDriverPermittedPeer *.syslog.nr-data.net
   
   *.* @@newrelic.syslog.nr-data.net:6514;NRLogFormat

   Copy

6. Restart the rsyslog service by running:

   bash

   Copy

   $
   sudo systemctl restart rsyslog

7. Check your New Relic account for logs.

Configure syslog-ng

To forward logs to New Relic with syslog-ng:

1. Install ca-certificates required to allow syslog-ng to send logs using TLS encryption:

   bash

   Copy

   $
   sudo yum install ca-certificates

2. Open the syslog-ng configuration file (/etc/syslog-ng/syslog-ng.conf) in a text editor.

3. Define the sources to be monitored by adding:

   source s_src {
           internal();
   };

   Copy

4. Optional: Configure syslog-ng to tail files by adding the following to the Sources configuration block:

   source s_files {
           file("PATH_TO_FILE");
   };

   Copy

5. Define the New Relic syslog format. For YOUR_LICENSE_KEY, use your New Relic license key.

   template NRFormat { template("YOUR_LICENSE_KEY <${PRI}>1 ${ISODATE} ${HOST:--} ${PROGRAM:--} ${PID:--} ${MSGID:--} ${SDATA:--} $MSG\n");
           template_escape(no);
   };

   Copy

6. Add the New Relic Syslog endpoint:

   destination d_newrelic {
         network("newrelic.syslog.nr-data.net" port(6514)
                transport("tls")
                tls(peer-verify(no))
                template(NRFormat)
         );
   };

   Copy

7. Add the following output to the log path configuration block:

   log {
          source(s_src);
          source(s_files);  ## in case step 4 is implemented.
          destination(d_newrelic);
   };

   Copy

8. Restart syslog-ng by running:

   bash

   Copy

   $
   sudo service syslog-ng restart

9. Check your New Relic account for logs.

Forwarders that doesn't allow to modify the syslog format

Some forwarders, CDNs, or platforms (like AirWatch) do not allow modification of the syslog format but do permit altering the message itself by appending or prepending data.

In such cases, you can use an alternative mechanism to pass the license key used to forward your logs. Simply append, prepend, or include nrLicenseKey=<YOUR_LICENSE_KEY> within the message. This method ensures that your logs are authenticated and forwarded to New Relic without needing to modify the syslog format.

For example, if your forwarder allows you to prepend data to the log message, you can configure it to add nrLicenseKey=<YOUR_LICENSE_KEY> at the beginning of each log message. Similarly, if appending data is allowed, you can add the license key at the end of each log message.

This approach is particularly useful for CDNs, hardware devices, or managed services where modifying the syslog format is not possible. By including the license key within the message, you ensure that your logs are properly authenticated and received by New Relic.

If you encounter any issues or need further assistance, refer to the documentation provided by your forwarder or CDN for specific instructions on how to append or prepend data to log messages.