Built-in log parsing rules

New Relic can parse common log formats according to built-in rules, so that you don't have to create your own parsing rules. Here are the log parsing rules, their Grok patterns, and what fields are parsed.

Parsing example

A good example is a default NGINX access log containing unstructured text. It's useful for searching but not much else. Here's an example of a typical line:

127.180.71.3 - - [10/May/1997:08:05:32 +0000] "GET /downloads/product_1 HTTP/1.1" 304 0 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.21)"

In an unparsed format, you would need to do a full text search to answer most questions. After parsing, the log is organized into attributes, like response code and request URL:

{
  "remote_addr": "93.180.71.3",
  "time": "1586514731",
  "method": "GET",
  "path": "/downloads/product_1",
  "version": "HTTP/1.1",
  "response": "304",
  "bytesSent": 0,
  "user_agent": "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.21)"
}

Parsing makes it easier to create custom queries that facet on those values. This helps you understand the distribution of response codes per request URL and quickly find problematic pages.

Add the `logtype` attribute

When aggregating logs, it's important to provide metadata that makes it easy to organize, search, and parse those logs. One simple way of doing this is to add the attribute logtype to the log messages when they're shipped. Built-in parsing rules are applied by default to certain logtype values.

Tip

The fields logType, logtype, and LOGTYPE are all supported for built-in rules. For ease of searching, we recommend that you align on a single syntax in your organization.

Here are some examples of how to add logtype to logs sent by some of our supported shipping methods.

Add logtype as an attribute. You must set the logtype for each named source.

logs:
  - name: file-simple
    file: /path/to/file
    attributes:
      logtype: fileRaw
  - name: nginx-example
    file: /var/log/nginx.log
    attributes:
      logtype: nginx

Add a filter block to the .conf file, which uses a record_transformer to add a new field. In this example we use a logtype of nginx to trigger the build-in NGINX parsing rule. Check out other Fluentd examples.

<filter containers>
  @type record_transformer
  enable_ruby true
  <record>
    #Add logtype to trigger a built-in parsing rule for nginx access logs
    logtype nginx
    #Set timestamp from the value contained in the field "time"
    timestamp record["time"]
    #Add hostname and tag fields to all records
    hostname "#{Socket.gethostname}"
    tag ${tag}
  </record>
</filter>

Add a filter block to the .conf file that uses a record_modifier to add a new field. In this example we use a logtype of nginx to trigger the build-in NGINX parsing rule. Check out other Fluent Bit examples.

[FILTER]
    Name   record_modifier
    Match  *
    Record logtype nginx
    Record hostname ${HOSTNAME}
    Record service_name Sample-App-Name

Add a filter block to the Logstash configuration which uses an add_field mutate filter to add a new field. In this example we use a logtype of nginx to trigger the build-in NGINX parsing rule. Check out other Logstash examples.

filter {
  mutate {
    add_field => {
      "logtype" => "nginx"
      "service_name" => "myservicename"
      "hostname" => "%{host}"
    }
  }
}

You can add attributes to the JSON request sent to New Relic. In this example we add a logtype attribute of value nginx to trigger the built-in NGINX parsing rule. Learn more about using the Logs API.

POST /log/v1 HTTP/1.1
Host: log-api.newrelic.com
Content-Type: application/json
X-License-Key: YOUR_LICENSE_KEY
Accept: */*
Content-Length: 133
{
  "timestamp": TIMESTAMP_IN_UNIX_EPOCH,
  "message": "User 'xyz' logged in",
  "logtype": "nginx",
  "service": "login-service",
  "hostname": "login.example.com"
}

To manage your parsing rules programmatically, use NerdGraph, our GraphQL-format API, at api.newrelic.com/graphiql. For more information, see the NerdGraph tutorial to create, query, and delete your parsing rules.

Source: logtype = 'apache'

Grok:

%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}

Results:

clientip: The IP address of the client.
verb: The HTTP verb
ident: The user identity of the client making the request
response: The HTTP status code of the response
request: The URI and request being made
httpversion: The HTTP version of the request
rawrequest: The raw HTTP request if data is posted
bytes: The number of bytes sent
referrer: The HTTP referrer
agent: The client's user agent

Source: logtype = 'apache_error'

Grok:

\[%{DATA:apache_error.timestamp}\] \[%{WORD:apache_error.source}:%{DATA:level}\] \[pid %{NUMBER:apache_error.pid}(:tid %{NUMBER:apache_error.tid})?\] (%{DATA:apache_error.sourcecode}\(%{NUMBER:apache_error.linenum}\): )?(?:\[client %{IPORHOST:apache_error.clientip}:%{POSINT:apache_error.port}\] ){0,1}%{GREEDYDATA:apache_error.message}

Results:

apache_error.timestamp: The timestamp of the log statement
apache_error.source: The source module
level: The log level
apache_error.pid: The apache PID (process identifier)
apache_error.tid: The apache TID (thread identifier)
apache_error.sourcecode: The apache sourcecode
apache_error.linenum: The sourcecode line number
apache_error.clientip: The client IP address
apache_error.port: The client IP port number
apache_error.message: The error message

Source: logtype = 'alb'

Grok:

^%{NOTSPACE:type} %{TIMESTAMP_ISO8601:time} %{NOTSPACE:elb} %{NOTSPACE:client_ip}:%{NOTSPACE:client_port} ((%{NOTSPACE:target_ip}:%{NOTSPACE:target_port})|-) %{NOTSPACE:request_processing_time} %{NOTSPACE:target_processing_time} %{NOTSPACE:response_processing_time} %{NOTSPACE:elb_status_code} %{NOTSPACE:target_status_code} %{NOTSPACE:received_bytes} %{NOTSPACE:sent_bytes} "%{DATA:request}" "%{DATA:user_agent}" %{NOTSPACE:ssl_cipher} %{NOTSPACE:ssl_protocol} %{NOTSPACE:target_group_arn} "%{DATA:trace_id}" "%{NOTSPACE:domain_name}" "%{NOTSPACE:chosen_cert_arn}" %{NOTSPACE:matched_rule_priority} %{TIMESTAMP_ISO8601:request_creation_time} "%{NOTSPACE:actions_executed}" "%{NOTSPACE:redirect_url}" "%{NOTSPACE:error_reason}" (?:"|)%{DATA:target_port_list}(?:"|) (?:"|)%{DATA:target_status_code_list}(?:"|) "%{NOTSPACE:classification}" "%{NOTSPACE:classification_reason}"

Results:

Field	Definition
`type`	The type of request or connection. Possible values are: `http`: HTTP `https`: HTTP over SSL/TLS `h2`: HTTP/2 over SSL/TLS `ws`: WebSockets `wss`: WebSockets over SSL/TLS
`elb`	The resource ID of the load balancer. If you are parsing access log entries, note that resources IDs can contain forward slashes (`/`).
`client_ip`	The IP address of the requesting client
`client_port`	The IP port of the requesting client
`target_ip`	The IP address of the target that processed this request. If the client didn't send a full request, the load balancer can't dispatch the request to a target, and this value is set to `-`. If the target is a Lambda function, this value is set to `-`. If the request is blocked by AWS WAF, this value is set to `-`, and the value of `elb_status_code` is set to `403`.
`target_port`	The IP port of the target that processed this request. If the client didn't send a full request, the load balancer can't dispatch the request to a target, and this value is set to `-`. If the target is a Lambda function, this value is set to `-`. If the request is blocked by AWS WAF, this value is set to `-`, and the value of `elb_status_code` is set to `403`.
`target_port_list`	The IP address and port of the target that processed this request. If the client didn't send a full request, the load balancer can't dispatch the request to a target, and this value is set to `-`. If the target is a Lambda function, this value is set to `-`. If the request is blocked by AWS WAF, this value is set to `-`, and the value of `elb_status_code` is set to `403`.
`request_processing_time`	The total time elapsed (in seconds, with millisecond precision) from the time the load balancer received the request until the time it sent it to a target. This value is set to `-1` if the load balancer can't dispatch the request to a target. This can happen if the target closes the connection before the idle timeout or if the client sends a malformed request. This value can also be set to `-1` if the registered target does not respond before the idle timeout.
`target_processing_time`	The total time elapsed (in seconds, with millisecond precision) from the time the load balancer sent the request to a target until the target started to send the response headers. This value is set to `-1` if the load balancer can't dispatch the request to a target. This can happen if the target closes the connection before the idle timeout or if the client sends a malformed request. This value can also be set to `-1` if the registered target does not respond before the idle timeout.
`response_processing_time`	The total time elapsed (in seconds, with millisecond precision) from the time the load balancer received the response header from the target until it started to send the response to the client. This includes both the queuing time at the load balancer and the connection acquisition time from the load balancer to the client. This value is set to `-1` if the load balancer can't send the request to a target. This can happen if the target closes the connection before the idle timeout or if the client sends a malformed request.
`elb_status_code`	The status code of the response from the load balancer
`target_status_code_list`	The status code of the response from the target. This value is recorded only if a connection was established to the target and the target sent a response. Otherwise, it is set to `-`.
`received_bytes`	The size of the request, in bytes, received from the client (requester). For HTTP requests, this includes the headers. For WebSockets, this is the total number of bytes received from the client on the connection.
`sent_bytes`	The size of the response, in bytes, sent to the client (requester). For HTTP requests, this includes the headers. For WebSockets, this is the total number of bytes sent to the client on the connection.
`request`	The HTTP request
`user_agent`	User-Agent string that identifies the client that originated the request, enclosed in double quotes. The string consists of one or more product identifiers, product/version. If the string is longer than 8 KB, it is truncated.
`ssl_cipher`	The SSL cipher. This value is set to `-` if the listener is not an HTTPS listener.
`ssl_protocol`	The SSL protocol. This value is set to `-` if the listener is not an HTTPS listener.
`target_group_arn`	The Amazon Resource Name (ARN) of the target group
`trace_id`	The contents of the `X-Amzn-Trace-Id header`, enclosed in double quotes
`domain_name`	The SNI domain provided by the client during the TLS handshake, enclosed in double quotes. This value is set to `-` if the client doesn't support SNI or the domain doesn't match a certificate and the default certificate is presented to the client.
`chosen_cert_arn`	The ARN of the certificate presented to the client, enclosed in double quotes. Set to `session-reused` if the session is reused. Set to `-` if the listener is not an HTTPS listener.
`matched_rule_priority`	The priority value of the rule that matched the request. If a rule matched, this is a value from `1` to `50000`. If no rule matched and the default action was taken, this value is set to `0`. If an error occurs during rules evaluation, it is set to `-1`. For any other error, it is set to `-`.
`request_creation_time`	The time when the load balancer received the request from the client, in ISO 8601 format.
`actions_executed`	The actions taken when processing the request, enclosed in double quotes. This value is a comma-separated list that can include the values described in `actions_taken`. If no action was taken, such as for a malformed request, this value is set to `-`.
`redirect_url`	The URL of the redirect target for the location header of the HTTP response, enclosed in double quotes. If no redirect actions were taken, this value is set to `-`.
`error_reason`	The error reason code, enclosed in double quotes. If the request failed, this is one of the error codes described in Error Reason Codes. If the actions taken do not include an authenticate action or the target is not a Lambda function, this value is set to `-`.
`classification`	Classification.
`classification_reason`	The classification reason.

Source: logtype = 'cassandra'

Grok

%{WORD:level}%{SPACE}\\[%{NOTSPACE:cassandra.thread}\\] %{TIMESTAMP_ISO8601:cassandra.timestamp} %{NOTSPACE:cassandra.source}:%{INT:cassandra.line.number} - %{GREEDYDATA:cassandra.message}

Results

level: The log level of the message (all, trace, debug, info, warn, error, off)
cassandra.thread: The name of the thread issuing the log statement
cassandra.timestamp: The timestamp of the log statement
cassandra.source: The source file name
cassandra.line.number: The line number of the message
cassandra.message: The raw cassandra message

Source: logtype = 'cloudfront-web'

Grok:

^%{NOTSPACE:date}%{SPACE}%{NOTSPACE:time}%{SPACE}%{NOTSPACE:x_edge_location}%{SPACE}%{NOTSPACE:sc_bytes}%{SPACE}%{NOTSPACE:c_ip}%{SPACE}%{NOTSPACE:cs_method}%{SPACE}%{NOTSPACE:cs_host}%{SPACE}%{NOTSPACE:cs_uri_stem}%{SPACE}%{NOTSPACE:sc_status}%{SPACE}%{NOTSPACE:cs_referer}%{SPACE}%{NOTSPACE:cs_user_agent}%{SPACE}%{NOTSPACE:cs_uri_query}%{SPACE}%{NOTSPACE:cs_Cookie}%{SPACE}%{NOTSPACE:x_edge_result_type}%{SPACE}%{NOTSPACE:x_edge_request_id}%{SPACE}%{NOTSPACE:x_host_header}%{SPACE}%{NOTSPACE:cs_protocol}%{SPACE}%{NOTSPACE:cs_bytes}%{SPACE}%{NOTSPACE:time_taken}%{SPACE}%{NOTSPACE:x_forwarded_for}%{SPACE}%{NOTSPACE:ssl_protocol}%{SPACE}%{NOTSPACE:ssl_cipher}%{SPACE}%{NOTSPACE:x_edge_response_result_type}%{SPACE}%{NOTSPACE:cs_protocol_version}%{SPACE}%{NOTSPACE:fle_status}%{SPACE}%{NOTSPACE:fle_encrypted_fields}%{SPACE}%{NOTSPACE:c_port}%{SPACE}%{NOTSPACE:time_to_first_byte}%{SPACE}%{NOTSPACE:x_edge_detailed_result_type}%{SPACE}%{NOTSPACE:sc_content_type}%{SPACE}%{NOTSPACE:sc_content_len}%{SPACE}%{NOTSPACE:sc_range_start}%{SPACE}%{NOTSPACE:sc_range_end}

Results:

Field	Definition
`x_edge_location`	The edge location that served the request. Each edge location is identified by a three-letter code and an arbitrarily assigned number; for example, `DFW3`. The three-letter code typically corresponds with the International Air Transport Association airport code for an airport near the edge location. (These abbreviations might change in the future.)
`sc_bytes`	The total number of bytes that CloudFront served to the viewer in response to the request, including headers; for example, `1045619`. For WebSocket connections, this is the total number of bytes sent from the server to the client through the connection.
`c_ip`	The IP address of the viewer that made the request, either in IPv4 or IPv6 format. If the viewer used an HTTP proxy or a load balancer to send the request, the value of `c_ip` is the IP address of the proxy or load balancer. See also `X-Forwarded-For`.
`cs_method`	The HTTP request method: `DELETE`, `GET`, `HEAD`, `OPTIONS`, `PATCH`, `POST`, or `PUT`.
`cs_host`	The domain name of the CloudFront distribution; for example, `d111111abcdef8.cloudfront.net`.
`cs_uri_stem`	The portion of the URI that identifies the path and object; for example, `/images/cat.jpg`. Question marks in URLs and query strings are not included.
`sc_status`	An HTTP status code; for example, `200`. Status code `000` indicates the viewer closed the connection (for example, closed the browser tab) before CloudFront could respond to a request. If the viewer closes the connection after CloudFront starts to send the response, the log contains the applicable HTTP status code.
`cs_referer`	The name of the domain that originated the request. Common referrers include search engines, other websites that link directly to your objects, and your own website.
`cs_user_agent`	The value of the User-Agent header in the request. The User-Agent header identifies the source of the request, such as the type of device and browser that submitted the request, and which search engine if applicable.
`cs_uri_query`	The query string portion of the URI, if any. When a URI doesn't contain a query string, this field's value is a hyphen (`-`).
`cs_cookie`	The cookie header in the request, including name-value pairs and the associated attributes. If you enable cookie logging, CloudFront logs the cookies in all requests regardless of which cookies you choose to forward to the origin. If a request doesn't include a cookie header, this field's value is a hyphen (`-`).
`x_edge_result_type`	How CloudFront classifies the response after the last byte left the edge location. In some cases, the result type can change between the time that CloudFront is ready to send the response and the time that CloudFront has finished sending the response.
`x_edge_request_id`	An encrypted string that uniquely identifies a request. In the response header, this is `x-amz-cf-id`.
`x_host_header`	The value that the viewer included in the Host header for this request. This is the domain name in the request. If you're using the CloudFront domain name in your object URLs, this field contains that domain name. If you're using alternate domain names in your object URLs, such as `http://example.com/logo.png`, this field contains the alternate domain name, such as `example.com`. To use alternate domain names, you must add them to your distribution.
`cs_protocol`	The protocol that the viewer specified in the request: `http`, `https`, `ws`, or `wss`.
`cs_bytes`	The number of bytes of data that the viewer included in the request, including headers. For WebSocket connections, this is the total number of bytes sent from the client to the server on the connection.
`time_taken`	The number of seconds (to the thousandth of a second; for example, 0.002) between the time that a CloudFront edge server receives a viewer's request and the time that CloudFront writes the last byte of the response to the edge server's output queue as measured on the server. From the perspective of the viewer, the total time to get the full object will be longer than this value due to network latency and TCP buffering.
`x_forwarded_for`	If the viewer used an HTTP proxy or a load balancer to send the request, the value of `c_ip` in field 5 is the IP address of the proxy or load balancer. In that case, this field is the IP address of the viewer that originated the request. This field contains IPv4 and IPv6 addresses as applicable. If the viewer did not use an HTTP proxy or a load balancer, the value of `x_forwarded_for` is a hyphen (`-`).
`ssl_protocol`	When `cs_protocol` in field 17 is `https`, this field contains the SSL/TLS protocol that the client and CloudFront negotiated for transmitting the request and response. Possible values include: SSLv3 TLSv1 TLSv1.1 TLSv1.2 When `cs_protocol` in field 17 is `http`, the value for this field is a hyphen (`-`).
`ssl_cipher`	When `cs_protocol` in field 17 is `https`, this field contains the SSL/TLS cipher that the client and CloudFront negotiated for encrypting the request and response. Possible values include: `ECDHE-RSA-AES128-GCM-SHA256` `ECDHE-RSA-AES128-SHA256` `ECDHE-RSA-AES128-SHA` `ECDHE-RSA-AES256-GCM-SHA384` `ECDHE-RSA-AES256-SHA384` `ECDHE-RSA-AES256-SHA` `AES128-GCM-SHA256` `AES256-GCM-SHA384` `AES128-SHA256` `AES256-SHA` `AES128-SHA` `DES-CBC3-SHA` `RC4-MD5` When `cs_protocol` in field 17 is `http`, the value for this field is a hyphen (`-`).
`x_edge_response_result_type`	How CloudFront classified the response just before returning the response to the viewer. Possible values include: `Hit`: CloudFront served the object to the viewer from the edge cache. `RefreshHit`: CloudFront found the object in the edge cache but it had expired, so CloudFront contacted the origin to verify that the cache has the latest version of the object. `Miss`: The request could not be satisfied by an object in the edge cache, so CloudFront forwarded the request to the origin server and returned the result to the viewer. `LimitExceeded`: The request was denied because a CloudFront limit was exceeded. `CapacityExceeded`: CloudFront returned a `503` error because the edge location didn't have enough capacity at the time of the request to serve the object. `Error`: Typically this means the request resulted in a client error (`sc_status` is `4xx`) or a server error (`sc_status` is `5xx`). If the value of `x_edge_result_type` is `Error` and the value of this field is not `Error`, the client disconnected before finishing the download. `Redirect`: CloudFront redirects from HTTP to HTTPS. If `sc_status` is `403` and you configured CloudFront to restrict the geographic distribution of your content, the request might have come from a restricted location.
`cs_protocol_version`	The HTTP version that the viewer specified in the request. Possible values include: `HTTP/0.9` `HTTP/1.0` `HTTP/1.1` `HTTP/2.0`
`fle_status`	When field-level encryption is configured for a distribution, this field contains a code that indicates whether the request body was successfully processed. If field-level encryption is not configured for the distribution, the value is a hyphen (`-`).
`fle-encrypted-fields`	The number of fields that CloudFront encrypted and forwarded to the origin. CloudFront streams the processed request to the origin as it encrypts data, so `fle-encrypted-fields` can have a value even if the value of `fle-status` is an error. If field-level encryption is not configured for the distribution, the value of `fle-encrypted-fields` is a hyphen (`-`).
`c_port`	The port number of the request from the viewer.
`time_to_first_byte`	The number of seconds between receiving the request and writing the first byte of the response, as measured on the server.
`x_edge_detailed_result_type`	When `x-edge-result-type` is not `Error`, this field contains the same value as `x-edge-result-type`. When `x-edge-result-type` is `Error`, this field contains the specific type of error.
`sc_content_type`	The value of the HTTP Content-Type header of the response.
`sc_content_len`	The value of the HTTP Content-Length header of the response.
`sc_range_start`	When the response contains the HTTP Content-Range header, this field contains the range start value.
`sc-range-end`	When the response contains the HTTP Content-Range header, this field contains the range end value.

Source: logtype = 'cloudfront-rtl'

Grok:

^%{NOTSPACE:timestamp}.\d{3}%{SPACE}%{NOTSPACE:c_ip}%{SPACE}%{NOTSPACE:time_to_first_byte}%{SPACE}%{NOTSPACE:sc_status}%{SPACE}%{NOTSPACE:sc_bytes}%{SPACE}%{NOTSPACE:cs_method}%{SPACE}%{NOTSPACE:cs_protocol}%{SPACE}%{NOTSPACE:cs_host}%{SPACE}%{NOTSPACE:cs_uri_stem}%{SPACE}%{NOTSPACE:cs_bytes}%{SPACE}%{NOTSPACE:x_edge_location}%{SPACE}%{NOTSPACE:x_edge_request_id}%{SPACE}%{NOTSPACE:x_host_header}%{SPACE}%{NOTSPACE:time_taken}%{SPACE}%{NOTSPACE:cs_protocol_version}%{SPACE}%{NOTSPACE:cs_ip_version}%{SPACE}%{NOTSPACE:cs_user_agent}%{SPACE}%{NOTSPACE:cs_referer}%{SPACE}%{NOTSPACE:cs_cookie}%{SPACE}%{NOTSPACE:cs_uri_query}%{SPACE}%{NOTSPACE:x_edge_response_result_type}%{SPACE}%{NOTSPACE:x_forwarded_for}%{SPACE}%{NOTSPACE:ssl_protocol}%{SPACE}%{NOTSPACE:ssl_cipher}%{SPACE}%{NOTSPACE:x_edge_result_type}%{SPACE}%{NOTSPACE:fle_encrypted_fields}%{SPACE}%{NOTSPACE:fle_status}%{SPACE}%{NOTSPACE:sc_content_type}%{SPACE}%{NOTSPACE:sc_content_len}%{SPACE}%{NOTSPACE:sc_range_start}%{SPACE}%{NOTSPACE:sc_range_end}%{SPACE}%{NOTSPACE:c_port}%{SPACE}%{NOTSPACE:x_edge_detailed_result_type}%{SPACE}%{NOTSPACE:c_country}%{SPACE}%{NOTSPACE:cs_accept_encoding}%{SPACE}%{NOTSPACE:cs_accept}%{SPACE}%{NOTSPACE:cache_behavior_path_pattern}%{SPACE}%{NOTSPACE:cs_headers}%{SPACE}%{NOTSPACE:cs_header_names}%{SPACE}%{NOTSPACE:cs_headers_count}$

Results:

Field	Definition
`timestamp`	The date and time at which the edge server finished responding to the request.
`c_ip`	The IP address of the viewer that made the request, for example, `192.0.2.183` or `2001:0db8:85a3:0000:0000:8a2e:0370:7334`. If the viewer used an HTTP proxy or a load balancer to send the request, the value of this field is the IP address of the proxy or load balancer. See also the `x-forwarded-for` field.
`time_to_first_byte`	The number of seconds between receiving the request and writing the first byte of the response, as measured on the server.
`sc_status`	An HTTP status code; for example, `200`. Status code `000` indicates the viewer closed the connection (for example, closed the browser tab) before CloudFront could respond to a request. If the viewer closes the connection after CloudFront starts to send the response, the log contains the applicable HTTP status code.
`sc_bytes`	The total number of bytes that the server sent to the viewer in response to the request, including headers. For WebSocket connections, this is the total number of bytes sent from the server to the client through the connection.
`cs_method`	The HTTP request method received from the viewer.
`cs_protocol`	The protocol of the viewer request (http, https, ws, or wss).
`cs_host`	The value that the viewer included in the Host header of the request. If you’re using the CloudFront domain name in your object URLs (such as d111111abcdef8.cloudfront.net), this field contains that domain name. If you’re using alternate domain names (CNAMEs) in your object URLs (such as www.example.com), this field contains the alternate domain name.
`cs_uri_stem`	The entire request URL, including the query string (if one exists), but without the domain name. For example, `/images/cat.webp?mobile=true`.
`cs_bytes`	The total number of bytes of data that the viewer included in the request, including headers. For WebSocket connections, this is the total number of bytes sent from the client to the server on the connection.
`x_edge_location`	The edge location that served the request. Each edge location is identified by a three-letter code and an arbitrarily assigned number (for example, DFW3). The three-letter code typically corresponds with the International Air Transport Association (IATA) airport code for an airport near the edge location’s geographic location. (These abbreviations might change in the future.)
`x_edge_request_id`	How CloudFront classifies the response after the last byte left the edge location. In some cases, the result type can change between the time that CloudFront is ready to send the response and the time that CloudFront has finished sending the response.
`x_edge_request_id`	An opaque string that uniquely identifies a request. CloudFront also sends this string in the x-amz-cf-id response header.
`x_host_header`	The domain name of the CloudFront distribution (for example, `d111111abcdef8.cloudfront.net`).
`time_taken`	The number of seconds (to the thousandth of a second, for example, 0.082) from when the server receives the viewer’s request to when the server writes the last byte of the response to the output queue, as measured on the server. From the perspective of the viewer, the total time to get the full response will be longer than this value because of network latency and TCP buffering.
`cs_protocol_version`	The HTTP version that the viewer specified in the request. Possible values include `HTTP/0.9`, `HTTP/1.0`, `HTTP/1.1`, `HTTP/2.0`, and `HTTP/3.0`.
`c_ip_version`	The IP version of the request (IPv4 or IPv6).
`cs_user_agent`	The value of the User-Agent header in the request. The User-Agent header identifies the source of the request, such as the type of device and browser that submitted the request or, if the request came from a search engine, which search engine.
`cs_referer`	The value of the Referer header in the request. This is the name of the domain that originated the request. Common referrers include search engines, other websites that link directly to your objects, and your own website.
`cs_cookie`	The Cookie header in the request, including name—value pairs and the associated attributes.
`cs_uri_query`	The query string portion of the request URL, if any.
`x_edge_response_result_type`	How the server classified the response just before returning the response to the viewer. See also the x-edge-result-type field. Possible values include: `Hit`: The server served the object to the viewer from the cache. `RefreshHit`: The server found the object in the cache but the object had expired, so the server contacted the origin to verify that the cache had the latest version of the object. `Miss`: The request could not be satisfied by an object in the cache, so the server forwarded the request to the origin server and returned the result to the viewer. `LimitExceeded`: The request was denied because a CloudFront quota (formerly referred to as a limit) was exceeded `CapacityExceeded`: The server returned a 503 error because it didn’t have enough capacity at the time of the request to serve the object. `Error`: Typically, this means the request resulted in a client error (the value of the sc-status field is in the 4xx range) or a server error (the value of the sc-status field is in the 5xx range). If the value of the x-edge-result-type field is Error and the value of this field is not Error, the client disconnected before finishing the download. `Redirect`: The server redirected the viewer from HTTP to HTTPS according to the distribution settings.
`x_forwarded_for`	If the viewer used an HTTP proxy or a load balancer to send the request, the value of the c-ip field is the IP address of the proxy or load balancer. In that case, this field is the IP address of the viewer that originated the request. This field contains an IPv4 address (for example, `192.0.2.183`) or an IPv6 address (for example, `2001:0db8:85a3:0000:0000:8a2e:0370:7334`).
`ssl_protocol`	When the request used HTTPS, this field contains the SSL/TLS protocol that the viewer and server negotiated for transmitting the request and response.
`ssl_cipher`	When the request used HTTPS, this field contains the SSL/TLS cipher that the viewer and server negotiated for encrypting the request and response.
`x_edge_result_type`	How the server classified the response after the last byte left the server. In some cases, the result type can change between the time that the server is ready to send the response and the time that it finishes sending the response. See also the `x-edge-response-result-type` field. For example, in HTTP streaming, suppose the server finds a segment of the stream in the cache. In that scenario, the value of this field would ordinarily be `Hit`. However, if the viewer closes the connection before the server has delivered the entire segment, the final result type (and the value of this field) is `Error`. WebSocket connections will have a value of `Miss` for this field because the content is not cacheable and is proxied directly to the origin. Possible values include: `Hit`: The server served the object to the viewer from the cache. `RefreshHit`: The server found the object in the cache but the object had expired, so the server contacted the origin to verify that the cache had the latest version of the object. `Miss`: The request could not be satisfied by an object in the cache, so the server forwarded the request to the origin and returned the result to the viewer. `LimitExceeded`: The request was denied because a CloudFront quota (formerly referred to as a limit) was exceeded. `CapacityExceeded`: The server returned an HTTP 503 status code because it didn’t have enough capacity at the time of the request to serve the object. `Error`: Typically, this means the request resulted in a client error (the value of the sc-status field is in the 4xx range) or a server error (the value of the sc-status field is in the 5xx range). If the value of the `sc-status` field is `200`, or if the value of this field is `Error` and the value of the `x-edge-response-result-type` field is not `Error`, it means the HTTP request was successful but the client disconnected before receiving all of the bytes. `Redirect`: The server redirected the viewer from HTTP to HTTPS according to the distribution settings.
`fle_encrypted_fields`	The number of field-level encryption fields that the server encrypted and forwarded to the origin. CloudFront servers stream the processed request to the origin as they encrypt data, so this field can have a value even if the value of `fle-status` is an error.
`fle_status`	When field-level encryption is configured for a distribution, this field contains a code that indicates whether the request body was successfully processed. When the server successfully processes the request body, encrypts values in the specified fields, and forwards the request to the origin, the value of this field is Processed. The value of `x-edge-result-type` can still indicate a client-side or server-side error in this case. Possible values for this field include: *`ForwardedByContentType`: The server forwarded the request to the origin without parsing or encryption because no content type was configured. `ForwardedByQueryArgs`: The server forwarded the request to the origin without parsing or encryption because the request contains a query argument that wasn’t in the configuration for field-level encryption. `ForwardedDueToNoProfile`: The server forwarded the request to the origin without parsing or encryption because no profile was specified in the configuration for field-level encryption. `MalformedContentTypeClientError`: The server rejected the request and returned an HTTP 400 status code to the viewer because the value of the Content-Type header was in an invalid format. `MalformedInputClientError`: The server rejected the request and returned an HTTP 400 status code to the viewer because the request body was in an invalid format. `MalformedQueryArgsClientError`: The server rejected the request and returned an HTTP 400 status code to the viewer because a query argument was empty or in an invalid format. `RejectedByContentType`: The server rejected the request and returned an HTTP 400 status code to the viewer because no content type was specified in the configuration for field-level encryption. `RejectedByQueryArgs`: The server rejected the request and returned an HTTP 400 status code to the viewer because no query argument was specified in the configuration for field-level encryption. `ServerError`: The origin server returned an error. If the request exceeds a field-level encryption quota (formerly referred to as a limit), this field contains one of the following error codes, and the server returns HTTP status code 400 to the viewer. `FieldLengthLimitClientError`: A field that is configured to be encrypted exceeded the maximum length allowed `FieldNumberLimitClientError`: A request that the distribution is configured to encrypt contains more than the number of fields allowed. `RequestLengthLimitClientError`: The length of the request body exceeded the maximum length allowed when field-level encryption is configured..
`sc_content_type`	The value of the HTTP Content-Type header of the response.
`sc_content_len`	The value of the HTTP Content-Length header of the response.
`sc-range-start`	When the response contains the HTTP Content-Range header, this field contains the range start value.
`sc-range-end`	When the response contains the HTTP Content-Range header, this field contains the range end value.
`c_port`	The port number of the request from the viewer.
`x_edge_detailed_result_type`	When the value of the `x-edge-result-type` field is `Error`, this field contains the specific type of error. When the object was served to the viewer from the Origin Shield cache, this field contains `OriginShieldHit`. In all other cases, this field contains the same value as `x-edge-result-type`. Possible values for this field include: `AbortedOrigin`: The server encountered an issue with the origin. `OriginCommError`: The request timed out while connecting to the origin, or reading data from the origin. `ClientCommError`: The response to the viewer was interrupted due to a communication problem between the server and the viewer. `ClientGeoBlocked`: The distribution is configured to refuse requests from the viewer’s geographic location. `ClientHungUpRequest`: The viewer stopped prematurely while sending the request. `Error` – An error occurred for which the error type doesn’t fit any of the other categories. This error type can occur when the server serves an error response from the cache. `InvalidRequest`: The server received an invalid request from the viewer. `InvalidRequestBlocked`: Access to the requested resource is blocked. `InvalidRequestCertificate`: The distribution doesn’t match the SSL/TLS certificate for which the HTTPS connection was established. `InvalidRequestHeader`: The request contained an invalid header. `InvalidRequestMethod`: The distribution is not configured to handle the HTTP request method that was used. This can happen when the distribution supports only cacheable requests. `OriginConnectError`: The server couldn’t connect to the origin. `OriginContentRangeLengthError`: The Content-Length header in the origin’s response doesn’t match the length in the Content-Range header. `OriginDnsError`: The server couldn’t resolve the origin’s domain name. `OriginError`: The origin returned an incorrect response. `OriginHeaderTooBigError`: A header returned by the origin is too big for the edge server to process. `OriginInvalidResponseError`: The origin returned an invalid response. `OriginReadError`: The server couldn’t read from the origin. `OriginShieldHit`: The object was served to the viewer from the Origin Shield cache. `OriginWriteError`: The server couldn’t write to the origin. `OriginZeroSizeObjectError`: A zero size object sent from the origin resulted in an error. `SlowReaderOriginError`: The viewer was slow to read the message that caused the origin error.
`c_country`	A country code that represents the viewer’s geographic location, as determined by the viewer’s IP address.
`cs_accept_encoding`	The value of the `Accept-Encoding` header in the viewer request.
`cs_accept`	The value of the `Accept` header in the viewer request.
`cache_behavior_path_pattern`	The path pattern that identifies the cache behavior that matched the viewer request.
`cs_headers`	The HTTP headers (names and values) in the viewer request.
`cs_header_names`	The names of the HTTP headers (not values) in the viewer request.
`cs_headers_count`	The number of HTTP headers in the viewer request.

Source: logtype = 'elb'

Grok:

^%{TIMESTAMP_ISO8601:time} %{NOTSPACE:elb} %{NOTSPACE:client_ip}:%{NOTSPACE:client_port} ((%{NOTSPACE:backend_ip}:%{NOTSPACE:backend_port})|-) %{NOTSPACE:request_processing_time} %{NOTSPACE:backend_processing_time} %{NOTSPACE:response_processing_time} %{NOTSPACE:elb_status_code} %{NOTSPACE:backend_status_code} %{NOTSPACE:received_bytes} %{NOTSPACE:sent_bytes} "%{DATA:request}" "%{DATA:user_agent}" %{NOTSPACE:ssl_cipher} %{NOTSPACE:ssl_protocol}

Results:

Field	Definition
`x_edge_location`	The edge location that served the request. Each edge location is identified by a three-letter code and an arbitrarily assigned number; for example, `DFW3`. The three-letter code typically corresponds with the International Air Transport Association airport code for an airport near the edge location. (These abbreviations might change in the future.)
`sc_bytes`	The total number of bytes that CloudFront served to the viewer in response to the request, including headers; for example, `1045619`. For WebSocket connections, this is the total number of bytes sent from the server to the client through the connection.
`c_ip`	The IP address of the viewer that made the request. If the viewer used an HTTP proxy or a load balancer to send the request, the value of `c_ip` is the IP address of the proxy or load balancer.
`cs_method`	The HTTP request method: `DELETE`, `GET`, `HEAD`, `OPTIONS`, `PATCH`, `POST`, or `PUT`.
`cs_host`	The domain name of the CloudFront distribution; for example, `d111111abcdef8.cloudfront.net`.
`cs_uri_stem`	The portion of the URI that identifies the path and object; for example, `/images/cat.jpg`. Question marks (`?`) in URLs and query strings are not included in the log.
`sc_status`	An HTTP status code (for example, `200`). Status code `000` indicates that the viewer closed the connection (for example, closed the browser tab) before CloudFront could respond to a request. If the viewer closes the connection after CloudFront starts to send the response, the log contains the applicable HTTP status code.
`cs_referer`	he name of the domain that originated the request. Common referrers include search engines, other websites that link directly to your objects, and your own website.
`cs_user_agent`	The value of the User-Agent header in the request. The User-Agent header identifies the source of the request, such as the type of device and browser that submitted the request and which search engine if applicable.
`cs_uri_query`	The query string portion of the URI, if any. When a URI doesn't contain a query string, this field's value is a hyphen (`-`).
`cs_cookie`	The cookie header in the request, including name-value pairs and the associated attributes. If you enable cookie logging, CloudFront logs the cookies in all requests, regardless of which cookies you choose to forward to the origin. If a request doesn't include a cookie header, this field's value is a hyphen (`-`).
`x_edge_result_type`	How CloudFront classifies the response after the last byte left the edge location. In some cases, the result type can change between the time that CloudFront is ready to send the response and the time that CloudFront has finished sending the response.
`x_edge_request_id`	An encrypted string that uniquely identifies a request. In the response header, this is `x-amz-cf-id`.
`x_host_header`	The value that the viewer included in the Host header for this request. This is the domain name in the request. If you're using the CloudFront domain name in your object URLs, this field contains that domain name. If you're using alternate domain names in your object URLs, such as `[http://example.com/logo.png`, this field contains the alternate domain name, such as `example.com`. To use alternate domain names, you must add them to your distribution.
`cs_protocol`	The protocol that the viewer specified in the request: `http`, `https`, `ws`, or `wss`.
`cs_bytes`	The number of bytes of data that the viewer included in the request, including headers. For WebSocket connections, this is the total number of bytes sent from the client to the server on the connection.
`time_taken`	The number of seconds (to the thousandth of a second; for example, 0.002) between the time that a CloudFront edge server receives a viewer's request and the time that CloudFront writes the last byte of the response to the edge server's output queue as measured on the server. From the perspective of the viewer, the total time to get the full object will be longer than this value due to network latency and TCP buffering.
`x_forwarded_for`	If the viewer used an HTTP proxy or a load balancer to send the request, the value of `c_ip` in field 5 is the IP address of the proxy or load balancer. In that case, this field is the IP address of the viewer that originated the request. This field contains IPv4 and IPv6 addresses as applicable. If the viewer did not use an HTTP proxy or a load balancer, the value of `x_forwarded_for` is a hyphen (`-`).
`ssl_protocol`	When `cs_protocol` in field 17 is `https`, this field contains the SSL/TLS protocol that the client and CloudFront negotiated for transmitting the request and response. Possible values include: SSLv3 TLSv1 TLSv1.1 TLSv1.2 When `cs_protocol` in field 17 is `http`, the value for this field is a hyphen (`-`).
`ssl_cipher`	When `cs_protocol` in field 17 is `https`, this field contains the SSL/TLS cipher that the client and CloudFront negotiated for encrypting the request and response. Possible values include: `ECDHE-RSA-AES128-GCM-SHA256` `ECDHE-RSA-AES128-SHA256` `ECDHE-RSA-AES128-SHA` `ECDHE-RSA-AES256-GCM-SHA384` `ECDHE-RSA-AES256-SHA384` `ECDHE-RSA-AES256-SHA` `AES128-GCM-SHA256` `AES256-GCM-SHA384` `AES128-SHA256` `AES256-SHA` `AES128-SHA` `DES-CBC3-SHA` `RC4-MD5` When `cs_protocol` is `http`, the value for this field is a hyphen (`-`).
`x_edge_response_result_type`	How CloudFront classified the response just before returning the response to the viewer. Possible values include: `Hit`: CloudFront served the object to the viewer from the edge cache. `RefreshHit`: CloudFront found the object in the edge cache but it had expired, so CloudFront contacted the origin to verify that the cache has the latest version of the object. `Miss`: The request could not be satisfied by an object in the edge cache, so CloudFront forwarded the request to the origin server and returned the result to the viewer. `LimitExceeded`: The request was denied because a CloudFront limit was exceeded. `CapacityExceeded`: CloudFront returned a `503` error because the edge location didn't have enough capacity at the time of the request to serve the object. `Error`: Typically this means the request resulted in a client error (`sc_status` is `4xx`) or a server error (`sc_status` is `5xx`). If the value of `x_edge_result_type` is `Error` and the value of this field is not `Error`, the client disconnected before finishing the download. `Redirect`: CloudFront redirects from HTTP to HTTPS. If `sc_status` is `403` and you configured CloudFront to restrict the geographic distribution of your content, the request might have come from a restricted location.
`cs_protocol_version`	The HTTP version that the viewer specified in the request. Possible values include: `HTTP/0.9` `HTTP/1.0` `HTTP/1.1` `HTTP/2.0`
`fle_status`	When field-level encryption is configured for a distribution, this field contains a code that indicates whether the request body was successfully processed. If field-level encryption is not configured for the distribution, the value of this field is a hyphen (`-`). When CloudFront successfully processes the request body, encrypts values in the specified fields, and forwards the request to the origin, the value of this field is `Processed`. The value of `x_edge_result_type` can still indicate a client-side or server-side error in this case. If the request exceeds a field-level encryption limit, `fle-status` contains one of the following error codes, and CloudFront returns HTTP status code `400` to the viewer.
`fle-encrypted-fields`	The number of fields that CloudFront encrypted and forwarded to the origin. CloudFront streams the processed request to the origin as it encrypts data, so `fle_encrypted_fields` can have a value even if the value of `fle_status` is an error. If field-level encryption is not configured for the distribution, the value of `fle_encrypted_fields` is a hyphen (`-`).
`c_port`	The port number of the request from the viewer.
`time_to_first_byte`	The number of seconds between receiving the request and writing the first byte of the response, as measured on the server.
`x_edge_detailed_result_type`	When `x_edge_result_type` is not `Error`, this field contains the same value as `x_edge_result_type`. When `x_edge_result_type` is `Error`, this field contains the specific type of error.
`sc_content_type`	The value of the HTTP Content-Type header of the response.
`sc_content_len`	The value of the HTTP Content-Length header of the response.
`sc_range_start`	When the response contains the HTTP Content-Range header, this field contains the range start value.
`sc-range-end`	When the response contains the HTTP Content-Range header, this field contains the range end value.

Source: logtype = 'haproxy_http'

Grok:

%{HOSTPORT:client} \\[%{NOTSPACE:haproxy_timestamp}\\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{NUMBER:time_queue}/%{NUMBER:time_backend_connect}/%{NUMBER:time_duration} %{NUMBER:bytes_read} %{NOTSPACE:termination_state} %{NUMBER:actconn}/%{NUMBER:feconn}/%{NUMBER:beconn}/%{NUMBER:srvconn}/%{NUMBER:retries} %{NUMBER:srv_queue}/%{NUMBER:backend_queue}

%{HOSTPORT:client} \\[%{NOTSPACE:haproxy_timestamp}\\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{NUMBER:time_client_req}/%{NUMBER:time_queue}/%{NUMBER:time_backend_connect}/%{NUMBER:time_server_response}/%{NUMBER:time_duration} %{NUMBER:status_code} %{NUMBER:bytes_read} %{NOTSPACE:captured_request_cookie} %{NOTSPACE:captured_response_cookie} %{NOTSPACE:termination_state_with_cookie_status} %{NUMBER:actconn}/%{NUMBER:feconn}/%{NUMBER:beconn}/%{NUMBER:srvconn}/%{NUMBER:retries} %{NUMBER:srv_queue}/%{NUMBER:backend_queue}?( \\\"%{GREEDYDATA:full_http_request}\\\")?( %{NOTSPACE:captured_response_headers})?

Results:

client: Source IP/Port of this request
haproxy_timestamp: Timestamp when this request was accepted
frontend_name: Name of the frontend used in this request
backend_name: Name of the backend used in this request
server_name: Server name in the backend group used in this request
time_client_req: Time waiting for full request from client (ms)
time_queue: Time waiting in queues (ms)
time_backend_connect: Time to establish connection to destination server (ms)
time_server_response: Time for destination server to send response (ms)
time_duration: Total time request active in HAProxy (ms)
status_code: HTTP response code
bytes_read: Total bytes read in this request
captured_request_cookie: Captured cookie from the request
captured_response_cookie: Captured cookie from the response
termination_state: Session state at disconnection
termination_state_with_cookie_status: Session state, including cookie status, at disconnection
actconn: Active connections
feconn: Frontend connections
beconn: Backend connections
srvconn: Server connections
retries: Retries
srv_queue: Server queue size
backend_queue: Backend queue size
full_http_request: The complete HTTP request line
captured_response_headers: Captured header from the response

Source: logtype = 'ktranslate-health'

Grok:

%{NOTSPACE:timestamp} ktranslate(/)?(%{GREEDYDATA:container_service})? \[%{NOTSPACE:severity}] %{GREEDYDATA:message}

Results:

timestamp: The time of the log
container_service: The unique identifier used to distinguish containers in ktranslate. This is set during Docker runtime with the optional --service_name flag
severity: The severity of the log line
message: The message field contains a free-form message that provides information about the event

Source: logtype = 'linux_cron'

Grok:

%{SYSLOGTIMESTAMP:linux_cron.timestamp} %{NOTSPACE:linux_cron.hostname} %{DATA:linux_cron.process}(\[%{NUMBER:linux_cron.pid:integer}\])?: (\(%{DATA:linux_cron.user}\))?%{GREEDYDATA:linux_cron.message}

Results:

linux_cron.timestamp: The time of the log
linux_cron.hostname: The linux server hostname
linux_cron.process: The linux cron process name
linux_cron.pid: The linux cron PID (process identifier)
linux_cron.user: The linux user that executed the cron
linux_cron.message: The log message

Source: logtype = 'linux_messages'

Grok:

%{SYSLOGTIMESTAMP:linux_messages.timestamp} %{NOTSPACE:linux_messages.hostname} %{DATA:linux_messages.process}(\[%{NUMBER:linux_messages.pid:integer}\])?: %{GREEDYDATA:linux_messages.message}

Results:

linux_messages.timestamp: The time of the log
linux_messages.hostname: The linux server hostname
linux_messages.process: The linux process name
linux_messages.pid: The linux PID (process identifier)
linux_messages.message: The log message

Source: logtype = 'iis_w3c'

Grok:

%{TIMESTAMP_ISO8601:log_timestamp} %{NOTSPACE:server_ip} %{WORD:method} %{NOTSPACE:uri} %{NOTSPACE:uri_query} %{NOTSPACE:server_port} %{NOTSPACE:username} %{NOTSPACE:client_ip} %{NOTSPACE:user_agent} %{NOTSPACE:referer} %{NOTSPACE:status} %{NOTSPACE:substatus} %{NOTSPACE:win32_status} %{NOTSPACE:time_taken}

Results:

IIS allows for several configuration options. If you've configured any options outside of the default, our Grok pattern won't parse your logs. In this case, we recommended that you use custom parsing.

Source: logtype = 'mongodb'

Grok:

%{TIMESTAMP_ISO8601:mongodb.timestamp} %{WORD:mongodb.severity} %{WORD:mongodb.component} *\[%{WORD:mongodb.context}\] %{GREEDYDATA:mongodb.message}

Results:

mongodb.timestamp: The timestamp of the log statement
mongodb.severity: The severity level of the log statement (F=Fatal, E=Error, W=Warning, I=Informational, D1-5=Debug)
mongodb.component: The category of the thread issuing the log statement
mongodb.context: The name of the thread issuing the log statement
mongodb.message: The raw mongodb message

Source: logtype = 'nginx'

Grok:

%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}

Results:

clientip: The IP address of the client
verb: The HTTP verb
ident: The user identity of the client making the request
response: The HTTP status code of the response
request: The URI and request being made
httpversion: The HTTP version of the request
rawrequest: The raw HTTP request if data is posted
bytes: The number of bytes sent
referrer: The HTTP referrer
agent: The client's user agent

Source: logtype = 'nginx-error'

Grok:

^(?<timestamp>%{YEAR:year}[./-]%{MONTHNUM:month}[./-]%{MONTHDAY:day}[- ]%{TIME:time}) \[%{WORD:severity}\] %{POSINT:pid}#%{NUMBER}: \*%{NUMBER} %{GREEDYDATA:errormessage} client: %{IPORHOST:client}, server: %{NOTSPACE:server}, request: (\\)?"%{DATA:request}", (?:, upstream: \"%{URI:upstream}\")?host: (\\)?"%{NOTSPACE:host}(\\)?"(, referrer: (\\)?"%{URI:referrer}(\\)?")?

Results:

severity: The severity of the log line
pid: The server process ID
errormessage: The error message
clientip: The IP address of the calling client
server: The server IP address
request: The full request
upstream: The upstream URI
host: The server's hostname
referrer: The HTTP referrer

Source: logtype = 'postgresql'

Grok:

%{DATA:postgresql.timestamp} \[%{NUMBER:postgresql.pid}\] %{WORD:level}:\s+%{GREEDYDATA:postgresql.message}

Results

postgresql.timestamp: The timestamp of the log
postgresql.pid: The server process ID
level: The log level of the message
postgresql.message: The log message

Source: logtype = 'rabbitmq'

Grok:

%{TIMESTAMP_ISO8601:rabbitmq.timestamp} \[%{LOGLEVEL:level}\] \<%{DATA:rabbitmq.pid}\> %{GREEDYDATA:rabbitmq.message}

Results:

rabbitmq.timestamp: The timestamp of the log
level: The log level of the message (debug, info, warning, error, critical, none)
rabbitmq.pid: The process id of the log line
rabbitmq.message: The rabbitmq error message

Source: logtype = 'redis'

Grok:

%{POSINT:redis.pid}:%{NOTSPACE:redis.role} (?<redistimestamp>[\d-]+ [a-zA-Z]+ [\d]+ [\d:]+.[\d]{3}) %{NOTSPACE:level} %{GREEDYDATA:redis.message}

Results:

redis.pid: The process id of the log line
redis.role: The role for the instance (X sentinel, C RDB/AOF writing child, S slave, M master)
redistimestamp: The timestamp of the log
level: The log level of the message (. debug, - verbose, * notice, # warning)
redis.message: The redis error message

Source: logtype = 'route-53'

Grok:

%{NUMBER:log_format_version} %{TIMESTAMP_ISO8601} %{WORD:zone_id} %{IPORHOST:query} %{WORD:query_type} %{WORD:response_code} %{WORD:protocol} %{WORD:edge_location} %{IP:resolver_ip} %{GREEDYDATA:edns_client_subnet}

Results:

log_format_version: A versioned format for the log.
zone_id: The ID of the hosted zone that is associated with all the DNS queries in this log.
query: The domain or subdomain that was specified in the request.
query_type: Either the DNS record type that was specified in the request, or ANY.
response_code: The DNS response code that Route 53 returned in response to the DNS query.
protocol: The protocol that was used to submit the query, either TCP or UDP.
edge_location: The Route 53 edge location that responded to the query. Each edge location is identified by a three-letter code and an arbitrary number; for example, DFW3. The three-letter code typically corresponds with the International Air Transport Association airport code for an airport near the edge location. (These abbreviations might change in the future.)
resolver_ip: The IP address of the DNS resolver that submitted the request to Route 53.
edns_client_subnet: A partial IP address for the client that the request originated from, if available from the DNS resolver.

Source: logtype = 'syslog-rfc5424'

Grok:

<%{NONNEGINT:pri}>%{NONNEGINT:version} +(?:%{TIMESTAMP_ISO8601:log.timestamp}|-) +(?:%{HOSTNAME:hostname}|-) +(?:\\-|%{NOTSPACE:app.name}) +(?:\\-|%{NOTSPACE:procid}) (?:\\-|%{NOTSPACE:msgid}) +(?:\[%{DATA:structured.data}\]|-|) +%{GREEDYDATA:message}

Results:

pri: The priority represents both the message facility and severity.
version: Syslog protocol version.
log.timestamp: Original timestamp.
hostname: The machine that originally sent the Syslog message.
app.name: The device or application that originated the message.
procid: The process name or process ID associated with a Syslog system.
msgid: Identifies the type of message.
structured.data: Structured data string value.
sd.sd-id.sd-param-name: The structured.data content is also parsed into separate attributes following a predefined naming convention: sd.sd-id.sd-param-name. See the structured data parsing examples, which follow.
message: Free-form message that provides information about the event.

Structured data parsing examples:

The structured data [example one="1" two="2"] would be parsed into two different attributes:

sd.example.one: "1"
sd.example.two: "2"

If the same structured data block contains duplicate param names, it also appends an index-based suffix on the attribute name. For example, the structured data [example number="1" number="2"] would be parsed as:

sd.example.number.0: "1"
sd.example.number.1: "2"

For structured data with enterprise numbers assigned, an extra attribute is also parsed. For example, the structured data [example@123 number="1"] would be parsed as:

sd.example.enterprise.number: 123
sd.example.number: "1"

Parsing example

Add the `logtype` attribute

Tip

New Relic infrastructure agent example

Fluentd example

Fluent Bit example

Logstash example

Logs API example

Apache

Apache Error

Application Load Balancer

Cassandra

Results

CloudFront (standard logs)

CloudFront (real-time logs)

Elastic Load Balancer

HAProxy

KTranslate Health

Linux Cron (/var/log/cron)

Linux Messages (/var/log/messages)

Microsoft IIS

Mongodb

Monit

MySQL Error

NGINX

NGINX Error

Postgresql

Results

Rabbitmq

Redis

Route 53

Syslog RFC-5424

Built-in log parsing rules

Parsing example .css-21sua1{background:none;border:none;width:0;padding:0;}

Add the logtype attribute

Tip

Fluentd example

Fluent Bit example

Logstash example

Logs API example

Apache

Apache Error

Application Load Balancer

Cassandra

CloudFront (standard logs)

CloudFront (real-time logs)

Elastic Load Balancer

HAProxy

KTranslate Health

Linux Cron (/var/log/cron)

Linux Messages (/var/log/messages)

Microsoft IIS

Mongodb

Monit

MySQL Error

NGINX

NGINX Error

Postgresql

Rabbitmq

Redis

Route 53

Syslog RFC-5424

Parsing example

Add the `logtype` attribute