Processors and their types

Processors operating within a Pipeline are responsible for executing precise data-structuring actions on logs. They aim to manipulate log data by generating additional attributes that enhance the logs with pertinent information.

Processors execute actions like parsing log lines using the Grok parser or severity remapper. These actions help extract valuable data from log entries, create new attributes based on the extracted information, and remap existing attributes to enhance the log data.

Grok Parser

The Grok parser uses predefined or custom patterns defined using regular expressions to match log lines and break them down into meaningful fields or attributes. It allows log data to be transformed from a raw text format into structured data, making it easier to analyze and search.

With a Grok parser, you can define patterns that match specific log formats or known patterns within log messages. For example, you can define patterns to extract timestamps, log levels, error codes, IP addresses, or any other relevant information from your logs.

Example:

# Sample Log Events:

[Sat Aug 12 04:05:51 2006] [notice] Apache/1.3.11 (Unix) mod_perl/1.21 -- configured resuming normal operations


# Grok Rules:

  \[%{DATA:logdate}\] \[%{DATA:status}\] %{DATA:source} --\ %{DATA:Message}

Let's break down the provided Grok parser rule and its sample log event:

%{DATA:logdate}: This part of the rule matches and captures the timestamp in the format "Sat Aug 12 04:05:51 2022" from the log event. The captured timestamp is assigned to the field "logdate".

%{DATA:severity}: Matches and captures any non-empty string of characters as the status. The captured value is assigned to the field "severity".

%{DATA:source}: Matches and captures an string as the source of the log. The captured value is assigned to the field "source".

%{DATA:os}: Matches and captures an string representing the which operating system used. The captured value is assigned to the field "os".

%{DATA:module}: Matches and captures an integer representing the number of sixes scored. The captured value is assigned to the field "Sixes".

%{DATA:Message}: Matches and captures any non-empty string of characters as the log message. The captured value is assigned to the field "Message".

Application to the Sample Log Event:

Using the provided Grok rule, we can apply it to the sample log event [Sat Aug 12 04:05:51 2006] [notice] Apache/1.3.11 (Unix) mod_perl/1.21 -- configured resuming normal operations to extract structured data:

Output Attributes:

  logdate: Sat Aug 12 04:05:51 2006
  severity: notice
  source: Apache/1.3.11 (Unix) mod_perl/1.21
  Message: resuming normal operations