Timestamp formats in logs

A log event leaves behind a timestamp attribute, which tells you the exact moment a certain event happened. Timestamps are critical for ordering events as they let you search logs for specific timeframes, letting you discover what and when an event happened.

Timestamps have many formats without a single standard. They appear at the beginning of the log event in most cases, but can sometimes appear much later in the log, or not at all.

By default, New Relic assigns the value of the log timestamp attribute (a reserved keyword) at the exact time the log is ingested. This is referred to as the ingest time.
When logs don't have a timestamp attribute, New Relic assigns a timestamp at time of ingest.
Logs in a JSON format contain some attribute that identifies the timestamp, like timestamp, log_timestamp, time, etc.
If we receive JSON logs with with a supported timestamp format, we override our ingest timestamp with the JSON attribute.

Supported logs timestamp format

There are two options for manually overriding default behavior to assign the timestamp value contained in the log.

If we receive a JSON formatted log containing a timestamp attribute in one of our supported time formats, the ingest timestamp will be overridden. Instead, the the timestamp value from the JSON attribute will be used.
You can use parsing rules to override the ingest timestamp value. Parsing rules let you extract out data to format your timestamp with one of our supported formats. You can also use datetime Grok type and a pattern to define the custom timestamp format in your log.

We support two timestamp formats:

The ISO8601 format follows the yyyy-MM-ddTHH:mm:ss.SSSTZD pattern (e.g. 2023-11-07T14:20:35+04:00). For example:

Key	Value	Example
yyyy-MM-dd	Four digit year Two digit month (01 for January, 02 for February, etc.) Two digit day (01 to 31)	2023-11-03
THH:mm:ss.SSS	`T` indicates start of the time element Two digit hour (00 through 23) Two digit minute (00 through 59) Two digit second (00 through 59) Three digit milliseconds (optional, up to nine digits with only the first three shown)	T14:20:35
TZD (timezone designator)	The `+` or `-` indicates how far ahead or behind a timezone is from UTC (Coordinated Universal Timie). UTC is assumed if not provided.	+04:00

Here are some format examples for ISO8601:

yyyy-MM-ddTHH:mm:ss: 2023-11-03T15:28:05
yyyy-MM-ddTHH:mm:ss.SSS: 2023-11-03T15:28:05.250
yyyy-MM-ddTHH:mm:ssZ: 2023-11-03T15:28:05Z
yyyy-MM-ddTHH:mm:ss+HH:mm: 2023-11-03T15:28:05+07:00
The maximum precision for timestamps is milliseconds. Timestamps expressed more precisely (e.g. nanoseconds or microseconds) automatically shorten to milliseconds.

Supported timestamps in JSON logs

If we receive a JSON formatted log with a timestamp attribute, we override the default ingest timestamp value with the original JSON value. If there's no timestamp field, or the field is incorrectly formatted, the log will be stored with the ingest timestamp. Keep in mind that a payload with timestamps older than 48 hours are dropped.

Timestamps are converted to Unix epoch milliseconds and stored internally as a long data type, with the reserved keyword timestamp. Depending on the JSON body format, the timestamp field can be allocated in four different places:

Inside the simplified set of attributes of the JSON body message when sending a single JSON object.
Inside the common object in the detailed set of attributes of the JSON body message when sending one or more JSON objects. The timestamp applies to all log messages of this JSON.
Inside each log message in the logs object in the detailed set of attributes of the JSON body message when sending one or more JSON objects. The timestamps only apply to that log message.
Inside the “message” JSON field when it is a valid JSON message. Our capabilities will parse any message attribute as JSON. The resulting JSON attributes in the parsed message will be added to the log.

Here are some examples of JSON logs with a valid timestamp attribute that override the ingest timestamp:

If we receive a JSON log like this with timestamp in epoch format (with no ms precision):

{
  "timestamp": 1701445572,
  "message": "This is my sample JSON log message with timestamp attribute in epoch format"
}

We will override the ingest timestamp with the epoch timestamp contained in the log, and the result will look like this:

"timestamp" : 1701445572000,
"message":"This is my sample JSON log message with timestamp attribute in epoch format"

If we receive a JSON log like this with timestamp in ISO8601 format like this:

{
  "timestamp": "2023-12-01T15:46:26.607Z",
  "message": "This is my sample JSON log message with timestamp attribute in ISO8601 format"
}

We will override the ingest timestamp with the ISO8601 timestamp (converted to epoch) contained in the log, and the result will look like this:

"timestamp": 1701445586607,
"message":"This is my sample JSON log message with timestamp attribute in ISO8601 format"

Set log timestamp with parsing rules

You can use parsing rules at ingest to override the ingest timestamp with the value contained in the log. Parsing rules extract out our supported time formats with simple Grok expressions, or by using the datetime Grok type and pattern to define the custom timestamp format in your log.

The attribute name extracted by the expression must be timestamp and its data type must be string or long. If data type is not provided, the timestamp defaults to string. For epoch timestamps with microsecond or nanosecond precision, you must include the long datatype even though timestamp precision shortens to milliseconds.

If the timestamp in your log is a Unix epoch or ISO8601 formatted timestamp you can use the default Grok patterns to match them. For example:

To extract a Unix epoch timestamp you can use any expression that matches the value, like NUMBER, NOTSPACE, DATA, or GREEDYDATA.
To extract an ISO8601 formatted timestamp you should use the TIMESTAMP_ISO8601 Grok expression.

You can use the datetime Grok type to parse timestamps in non-supported formats. To do this you must include the datetime Grok type with the pattern matching your logs timestamp format. The pattern must use the Java Simple Date and Time Patterns found here. Below are examples that show how Grok expressions transform unsupported timestamp formats into supported formats:

`timestamp`	`pattern`	Grok expression
11/17/2023 23:55:25.674	MM/dd/yyyy HH:mm:ss.SSS	`%{DATA:timestamp:datetime;MM/dd/yyyy HH:mm:ss.SSS}`
11-17-2023 23:55:25	MM-dd-yyyy HH:mm:ss	`%{DATA:timestamp:datetime;MM-dd-yyyy HH:mm:ss}`
11/17/2023 09:55:25 AM	MM/dd/yyyy h:mm:ss a	`%{DATA:timestamp:datetime;MM/dd/yyyy h:mm:ss a}`

Below are some examples on how to parse the timestamp from your logs using ingest Parsing rules:

Add this parsing rule:

%{TIMESTAMP_ISO8601:timestamp}

To this log message with timestamp in ISO8601 format:

2023-12-05T16:50:48.421Z 146.190.212.184: This is my sample non-JSON log message with ISO8601 timestamp.

To parse your log as follows:

"ISO8601_TIMEZONE": "Z",
"message": "2023-12-05T16:50:48.421Z 146.190.212.184: This is my sample non-JSON log message with ISO8601 timestamp.",
"timestamp": 1701795048421

Add the following parsing rule:

%{NUMBER:timestamp:long}

To the following log message with timestamp in epoch format:

1701795958504 146.190.212.184: This is my sample non-JSON log message with epoch (with ms) timestamp.

To parse your log as follows:

"message": "1701795958504 146.190.212.184: This is my sample non-JSON log message with epoch (with ms) timestamp.",
"timestamp": 1701795958504

Add this parsing rule:

^%{DATA:timestamp:datetime;MM/dd/yyyy HH:mm:ss.SSS} %{IP:ip}

To this log message with an un-supported timestamp format:

12/05/2023 17:12:45.347 146.190.212.184: This is my sample non-JSON log message with un-supported timestamp.

To parse your log as follows:

"ip": "146.190.212.184",
"message": "12/05/2023 17:12:45.347 146.190.212.184: This is my sample non-JSON log message with DATE_US timestamp.",
"timestamp": 1701796365347

Supported logs timestamp format

Unix epoch timestamp format

ISO8601 timestamp format

Supported timestamps in JSON logs

JSON log in epoch format (no ms precision)

JSON log in ISO8601 format

Set log timestamp with parsing rules

Parse rule for timestamps in ISO8601 format

Parse rule for timestamps in epoch format

Parse rule for timestamps in unsupported format

Timestamp formats in logs

Supported logs timestamp format .css-21sua1{background:none;border:none;width:0;padding:0;}

ISO8601 timestamp format

Supported timestamps in JSON logs

JSON log in epoch format (no ms precision)

JSON log in ISO8601 format

Set log timestamp with parsing rules

Parse rule for timestamps in ISO8601 format

Parse rule for timestamps in epoch format

Parse rule for timestamps in unsupported format

Supported logs timestamp format