Find data in long logs (blobs)

Extensive log data can help you troubleshoot issues. But what if an attribute in your log contains thousands of characters? How much of this data can New Relic store? And how can you find useful information in all this data?

How blobs work

For lengthy string values that are longer than can be stored in NRDB (4,094 characters), we store the long string in three pieces:

Long log sections	Description
First 4,094 characters	The first 4,094 characters are stored in a `Log` event field with the same name. So a long `message` value would have its first 4,094 characters stored in a `message` field.
Next 128,000 UTF-8 bytes	The next 128,000 UTF-8 bytes of the string are stored in a `blob` field with the name with `newrelic.ext.` prepended. So a long `message` value would have characters past the first 4,094 characters stored in a `newrelic.ext.message` field as a `blob`. The actual number of characters stored depends on the UTF-8 representation of the characters. UTF-8 represents Unicode characters as one to four bytes, so we will store anywhere between 32,000 and 128,000 characters past the first 4,094 characters.
Remaining characters	Any characters past 4,094 characters plus 128,000 bytes are dropped and not stored.
Query results	Blob queries have a limit of 20 results. When using `blob()` ensure that your query returns a maximum of 20 results.

So the long message field would be stored as:

message: <first 4,094 characters as a string>
newrelic.ext.message: <next 128,000 bytes as a 'blob'>

Tip

You can search the first 4,094 characters of a string attribute. You can also create alerts for the first 4,094 characters. However, since blob storage is not searchable, text beyond the first 4,094 characters is not searchable or alertable.

Query your data for blobs

one.newrelic.com > All capabilities > Logs: To query extended blob data in your logs, be sure to include backticks in your attribute's blob syntax.

To query for any log data in New Relic, run the following query:

SELECT * FROM Log

To expand the blob data, run the following query, using message or any other attribute. Be sure to enclose the blob's attribute with backticks. For example:

SELECT message, another-attribute, blob(`newrelic.ext.message`), blob(`newrelic.ext.another-attribute`) FROM Log

This expands the data in the blob so you can see (but not search) it. For example, New Relic returns:

{
  "message": <first 4,094 characters>
  "newrelic.ext.message": <the next 128,000 bytes as Base64> 
  "another-attribute": <first 4,094 characters>
  "newrelic.ext.another-attribute": <the next 128,000 bytes as Base64>
}

The logs UI automatically stitches the original value back together when looking at the Log Detail View. When querying using NRQL directly, you need to manually stitch the information together by:

Decoding the Base64 of the newrelic.ext. attribute value
Converting the resulting UTF-8 into a string
Appending that string to the first 4,094 characters in the "main" attribute

Data retention for long logs

NRDB retains your blob records for a month. If you have existing long log messages stored as LogExtendedRecord, that data will also continue to be available for a month in NRDB.

After a month passes, no more new LogExtendedRecord attributes will be created. They will all be stored in NRDB as blobs.

How blobs work .css-21sua1{background:none;border:none;width:0;padding:0;}

Tip

Query your data for blobs

Data retention for long logs

How blobs work