Logstash - Supported Outputs

Logstash provides multiple Plugins to support various data stores or search engines. The output events of logs can be sent to an output file, standard output or a search engine pke Elasticsearch. There are three types of supported outputs in Logstash, which are −

Standard Output

File Output

Null Output

Let us now discuss each of these in detail.

Standard Output (stdout)

It is used for generating the filtered log events as a data stream to the command pne interface. Here is an example of generating the total duration of a database transaction to stdout.

logstash.conf

This config file contains a stdout output plugin to write the total sql_duration to a standard output.

input {
   file {
      path => "C:/tpwork/logstash/bin/log/input.log"
   }
} 
filter {
   grok {
      match => [
         "message", "%{LOGLEVEL:loglevel} - %{NOTSPACE:taskid}
            - %{NOTSPACE:logger} - %{WORD:label}( - %{INT:duration:int})?" 
      ]
   }
   if [logger] == "TRANSACTION_START" {
      aggregate {
         task_id => "%{taskid}"
         code => "map[ sql_duration ] = 0"
         map_action => "create"
      }
   }
   if [logger] == "SQL" {
      aggregate {
         task_id => "%{taskid}"
         code => "map[ sql_duration ] ||= 0 ;
            map[ sql_duration ] += event.get( duration )"
      }
   }
   if [logger] == "TRANSACTION_END" {
      aggregate {
         task_id => "%{taskid}"
         code => "event.set( sql_duration , map[ sql_duration ])"
         end_of_task => true
         timeout => 120
      }
   }
}
output {
   if [logger] == "TRANSACTION_END" {
      stdout {
         codec => pne{format => "%{sql_duration}"}
      }
   }
}

Note − Please install the aggregate filter, if not installed already.

>logstash-plugin install Logstash-filter-aggregate

Run Logstash

We can run Logstash by using the following command.

>logstash –f logsatsh.conf

Input.log

The following code block shows the input log data.

INFO - 48566 - TRANSACTION_START - start
INFO - 48566 - SQL - transaction1 - 320
INFO - 48566 - SQL - transaction1 - 200
INFO - 48566 - TRANSACTION_END – end

stdout (it will be command prompt in windows or terminal in UNIX)

This is the total sql_duration 320 + 200 = 520.

520

File Output

Logstash can also store the filter log events to an output file. We will use the above-mentioned example and store the output in a file instead of STDOUT.

logstash.conf

This Logstash config file direct Logstash to store the total sql_duration to an output log file.

input {
   file {
      path => "C:/tpwork/logstash/bin/log/input1.log"
   }
} 
filter {
   grok {
      match => [
         "message", "%{LOGLEVEL:loglevel} - %{NOTSPACE:taskid} -
            %{NOTSPACE:logger} - %{WORD:label}( - %{INT:duration:int})?" 
      ]
   }
   if [logger] == "TRANSACTION_START" {
      aggregate {
         task_id => "%{taskid}"
         code => "map[ sql_duration ] = 0"
         map_action => "create"
      }
   }
   if [logger] == "SQL" {
      aggregate {
         task_id => "%{taskid}"
         code => "map[ sql_duration ] ||= 0 ;
            map[ sql_duration ] += event.get( duration )"
      }
   }
   if [logger] == "TRANSACTION_END" {
      aggregate {
         task_id => "%{taskid}"
         code => "event.set( sql_duration , map[ sql_duration ])"
         end_of_task => true
         timeout => 120
      }
   }
}
output {
   if [logger] == "TRANSACTION_END" {
      file {
         path => "C:/tpwork/logstash/bin/log/output.log"
         codec => pne{format => "%{sql_duration}"}
      }
   }
}

Run logstash

We can run Logstash by using the following command.

>logstash –f logsatsh.conf

input.log

The following code block shows the input log data.

INFO - 48566 - TRANSACTION_START - start
INFO - 48566 - SQL - transaction1 - 320
INFO - 48566 - SQL - transaction1 - 200
INFO - 48566 - TRANSACTION_END – end

output.log

The following code block shows the output log data.

520

Null Output

This is a special output plugin, which is used for analyzing the performance of input and filter Plugins.