Logstash Tutorial
Logstash Input Stage
Logstash Parse and Transform
Logstash Output Stage
Logstash Advanced Topics
Logstash Useful Resources
Selected Reading
- Logstash - Internal Architecture
- Logstash - Installation
- Logstash - ELK Stack
- Logstash - Introduction
- Logstash - Home
Logstash Input Stage
Logstash Parse and Transform
Logstash Output Stage
Logstash Advanced Topics
Logstash Useful Resources
Selected Reading
- Who is Who
- Computer Glossary
- HR Interview Questions
- Effective Resume Writing
- Questions and Answers
- UPSC IAS Exams Notes
Logstash - Transforming the Logs
Logstash - Transforming the Logs
Logstash offers various plugins to transform the parsed log. These plugins can Add, Delete, and Update fields in the logs for better understanding and querying in the output systems.
We are using the Mutate Plugin to add a field name user in every pne of the input log.
Install the Mutate Filter Plugin
To install the mutate filter plugin; we can use the following command.
>Logstash-plugin install Logstash-filter-mutate
logstash.conf
In this config file, the Mutate Plugin is added after the Aggregate Plugin to add a new field.
input { file { path => "C:/tpwork/logstash/bin/log/input.log" } } filter { grok { match => [ "message", "%{LOGLEVEL:loglevel} - %{NOTSPACE:taskid} - %{NOTSPACE:logger} - %{WORD:label}( - %{INT:duration:int})?" ] } if [logger] == "TRANSACTION_START" { aggregate { task_id => "%{taskid}" code => "map[ sql_duration ] = 0" map_action => "create" } } if [logger] == "SQL" { aggregate { task_id => "%{taskid}" code => "map[ sql_duration ] ||= 0 ; map[ sql_duration ] += event.get( duration )" } } if [logger] == "TRANSACTION_END" { aggregate { task_id => "%{taskid}" code => "event.set( sql_duration , map[ sql_duration ])" end_of_task => true timeout => 120 } } mutate { add_field => {"user" => "tutorialspoint.com"} } } output { file { path => "C:/tpwork/logstash/bin/log/output.log" } }
Run Logstash
We can run Logstash by using the following command.
>logstash –f logstash.conf
input.log
The following code block shows the input log data.
INFO - 48566 - TRANSACTION_START - start INFO - 48566 - SQL - transaction1 - 320 INFO - 48566 - SQL - transaction1 - 200 INFO - 48566 - TRANSACTION_END - end
output.log
You can see that there is a new field named “user” in the output events.
{ "path":"C:/tpwork/logstash/bin/log/input.log", "@timestamp":"2016-12-25T19:55:37.383Z", "@version":"1", "host":"wcnlab-PC", "message":"NFO - 48566 - TRANSACTION_START - start ", "user":"tutorialspoint.com","tags":["_grokparsefailure"] } { "duration":320,"path":"C:/tpwork/logstash/bin/log/input.log", "@timestamp":"2016-12-25T19:55:37.383Z","loglevel":"INFO","logger":"SQL", "@version":"1","host":"wcnlab-PC","label":"transaction1", "message":" INFO - 48566 - SQL - transaction1 - 320 ", "user":"tutorialspoint.com","taskid":"48566","tags":[] } { "duration":200,"path":"C:/tpwork/logstash/bin/log/input.log", "@timestamp":"2016-12-25T19:55:37.399Z","loglevel":"INFO", "logger":"SQL","@version":"1","host":"wcnlab-PC","label":"transaction1", "message":" INFO - 48566 - SQL - transaction1 - 200 ", "user":"tutorialspoint.com","taskid":"48566","tags":[] } { "sql_duration":520,"path":"C:/tpwork/logstash/bin/log/input.log", "@timestamp":"2016-12-25T19:55:37.399Z","loglevel":"INFO", "logger":"TRANSACTION_END","@version":"1","host":"wcnlab-PC","label":"end", "message":" INFO - 48566 - TRANSACTION_END - end ", "user":"tutorialspoint.com","taskid":"48566","tags":[] }Advertisements