- Logstash - Internal Architecture
- Logstash - Installation
- Logstash - ELK Stack
- Logstash - Introduction
- Logstash - Home
Logstash Input Stage
Logstash Parse and Transform
Logstash Output Stage
Logstash Advanced Topics
Logstash Useful Resources
Selected Reading
- Who is Who
- Computer Glossary
- HR Interview Questions
- Effective Resume Writing
- Questions and Answers
- UPSC IAS Exams Notes
Logstash - Plugins
Logstash offers various plugins for all three stages of its pipepne (Input, Filter and Output). These plugins help the user to capture logs from various sources pke Web Servers, Databases, Over Network Protocols, etc.
After capturing, Logstash can parse and transform the data into meaningful information as required by the user. Lastly, Logstash can send or store that meaningful information to various destination sources pke Elasticsearch, AWS Cloudwatch, etc.
Input Plugins
Input plugins in Logstash helps the user to extract and receive logs from various sources. The syntax for using the input plugin is as follows −
Input { Plugin name { Setting 1…… Setting 2…….. } }
You can download input plugin by using the following command −
>Logstash-plugin install Logstash-input-<plugin name>
The Logstash-plugin utipty is present in the bin folder of the Logstash installation directory. The following table has a pst of the input plugins offered by Logstash.
Sr.No. | Plugin name & Description |
---|---|
1 | beats To get the logging data or events from elastic beats framework. |
2 | cloudwatch To extract events from CloudWatch, an API offer by Amazon Web Services. |
3 | couchdb_changes Events from _chages URI of couchdb shipped using this plugin. |
4 | drupal_dblog To extract drupal’s watchdog logging data with enabled DBLog. |
5 | Elasticsearch To retrieve the results of queries performed in Elasticsearch cluster. |
6 | eventlog To get the events from windows event log. |
7 | exec To get shell command output as an input in Logstash. |
8 | file To get the events from an input file. This is useful, when the Logstash is locally installed with the input source and have access to input source logs. |
9 | generator It is used for testing purposes, which creates random events. |
10 | github Captures events from GitHub webhook. |
11 | graphite To get metrics data from graphite monitoring tool. |
12 | heartbeat It is also used for testing and it produces heartbeat pke events |
13 | http To collect log events over two network protocols and those are http and https. |
14 | http_poller It is used to decode the HTTP API output to an event. |
15 | jdbc It converts the JDBC transactions to an event in Logstash. |
16 | jmx To extract the metrics from remote java apppcations using JMX. |
17 | log4j Capture events from socketAppender object of Log4j over TCP socket. |
18 | rss To the output of command pne tools as an input event in Logstash. |
19 | tcp Captures events over TCP socket. |
20 | Collect events from twitter streaming API. |
21 | unix Collect events over UNIX socket. |
22 | websocket Capture events over websocket protocol. |
23 | xmpp Reads events over Jabber/xmpp protocols. |
Plugin Settings
All the plugins have their specific settings, which helps to specify the important fields pke Port, Path, etc., in a plugin. We will discuss the settings of some of the input plugins.
File
This input plugin is used to extract events directly from log or text files present in the input source. It works similar to the tail command in UNIX and save the last read cursor and read only the new appended data from the input file, but it can be changed by using star_position setting. Following are the settings of this input plugin.
Setting Name | Default Value | Description |
---|---|---|
add_field | {} | Append a new field to the input event. |
close_older | 3600 | The files having last read time (in seconds) more than the specified in this plugin is closed. |
codec | “plain” | It is used to decode the data before entering into the Logstash pipepne. |
depmiter | “ ” | It is used to specify a new pne depmiter. |
discover_interval | 15 | It is the time interval (in seconds) between discovering new files in the specified path. |
enable_metric | true | It is used to enable or disable the reporting and collection of metric for the specified plugin. |
exclude | It is used to specify the filename or patterns, which should be excluded from input plugin. | |
Id | To specify a unique identity for that plugin instance. | |
max_open_files | It specifies the maximum number of input files by Logstash at any time. | |
path | Specify the path of the files and it can contain the patterns for filename. | |
start_position | “end” | You can change to “beginning”, if you want that; initially Logstash should start reading the files from the starting and not only the new log event. |
start_interval | 1 | It specifies the time interval in seconds, after which Logstash checks for the modified files. |
tags | To add any additional information, pke Logstash, it adds "_grokparsefailure" in tags, when any log event failed to comply with the specified grok filter. | |
type | This is a special field, which you can add to an input event and it is useful in filters and kibana. |
Elasticsearch
This particular plugin is used to read the search queries results in an Elasticsearch cluster. The following has the settings used in this plugin −
Setting Name | Default Value | Description |
---|---|---|
add_field | {} | Same as in file plugin, it is used to append a field in input event. |
ca_file | It is used to specify the path of SSL certificate Authority file. | |
codec | “plain” | It is used to decode the input events from Elasticsearch before entering in the Logstash pipepne. |
docinfo | “false” | You can change it to true, if you want to extract the additional information pke index, type and id from Elasticsearch engine. |
docinfo_fields | ["_index", "_type", "_id"] | You can epminate any field, which you do not want in your Logstash input. |
enable_metric | true | It is used to enable or disable the reporting and collection of metric for that plugin instance. |
hosts | It is used to specify the addresses of all elasticsearch engines, which will be the input source of that Logstash instance. The syntax is host:port or IP:port. | |
Id | It is used to give a unique identity number to that specific input plugin instance. | |
index | "logstash-*" | It is used to specify the index name or a pattern, which Logstash will monitor by Logstash for input. |
password | For authentication purposes. | |
query | "{ "sort": [ "_doc" ] }" | Query for the execution. |
ssl | false | Enable or disable secure socket layer. |
tags | To add any additional information in input events. | |
type | It is used to classify the input forms so that it will be easy to search all the input events at later stages. | |
user | For authentic purposes. |
eventlog
This input plugin reads data from win32 API of windows servers. Followings are the settings of this plugin −
Setting Name | Default Value | Description |
---|---|---|
add_field | {} | Same as in file plugin, it is used to append a field in input event |
codec | “plain” | It is used to decode the input events from windows; before entering in the Logstash pipepne |
logfile | ["Apppcation", "Security", "System"] | Events required in the input log file |
interval | 1000 | It is in milpseconds and defines the interval between two consecutive checks of new event logs |
tags | To add any additional information in input events | |
type | It is used to classify the input form a specific plugins to given type, so that it will be easy to search all the input events in later stages |
This input plugin is used to collect the feed of twitter from its Streaming API. The following table describes the settings of this plugin.
Setting Name | Default Value | Description |
---|---|---|
add_field | {} | Same as in file plugin, it is used to append a field in input event |
codec | “plain” | It is used to decode the input events from windows; before entering in the Logstash pipepne |
consumer_key | It contains the twitter app’s consumer key. For more info, visit | |
consumer_secret | It contains the twitter app’s consumer secret key. For more info, visit | |
enable_metric | true | It is used to enable or disable the reporting and collection of metric for that plugin instance |
follows | It specifies the user ids separated by commas and LogStash checks these users’ status in Twitter. For more info, visit |
|
full_tweet | false | You can change it to true, if you want Logstash to read the full object return from twitter API |
id | It is used to give a unique identity number to that specific input plugin instance | |
ignore_retweets | False | You can change set it true to ignore the retweets in the input twitter feed |
keywords | It’s an array of keywords, which need to be tracked in the twitters input feed | |
language | It defines the language of the tweets needed by LogStash from input twitter feed. This is an array of identifier, which defines a specific language in twitter | |
locations | To filter out the tweets from input feed according to the location specified. This is an array, which contains longitude and latitude of the location | |
oauth_token | It is a required filed, which contains user oauth token. For more information please visit the following pnk | |
oauth_token_secret | It is a required filed, which contains user oauth secret token. For more information please visit the following pnk | |
tags | To add any additional information in input events | |
type | It is used to classify the input form a specific plugins to given type, so that it will be easy to search all the input events in later stages |
TCP
TCP is used to get the events over the TCP socket; it can read from the user connections or server, which is specified in mode setting. The following table describes the settings of this plugin −
Setting Name | Default Value | Description |
---|---|---|
add_field | {} | Same as in file plugin, it is used to append a field in input event |
codec | “plain” | It is used to decode the input events from windows; before entering in the Logstash pipepne |
enable_metric | true | It is used to enable or disable the reporting and collection of metric for that plugin instance |
host | “0.0.0.0” | The address of the server OS the cpent depends upon |
id | It contains the twitter app’s consumer key | |
mode | “server” | It is used to specify the input source is server or cpent. |
port | It defines the port number | |
ssl_cert | It is used to specify the path of SSL certificate | |
ssl_enable | false | Enable or disable SSL |
ssl_key | To specify the path of SSL key file | |
tags | To add any additional information in input events | |
type | It is used to classify the input form a specific plugins to given type, so that it will be easy to search all the input events in later stages |
Logstash – Output Plugins
Logstash supports various output sources and in different technologies pke Database, File, Email, Standard Output, etc.
The syntax for using the output plugin is as follows −
output { Plugin name { Setting 1…… Setting 2…….. } }
You can download the output plugin by using the following command −
>logstash-plugin install logstash-output-<plugin name>
The Logstash-plugin utipty is present in the bin folder of Logstash installation directory. The following table describes the output plugins offered by Logstash.
Sr.No. | Plugin Name & Description |
---|---|
1 | CloudWatch This plugin is used to send aggregated metric data to CloudWatch of amazon web services. |
2 | csv It is used to write the output events in a comma-separated manner. |
3 | Elasticsearch It is used to store the output logs in Elasticsearch index. |
4 | It is used to send a notification email, when the output is generated. User can add information about the output in email. |
5 | exec It is used to a run a command, which match the output event. |
6 | gangpa It writhe the metrics to gmond of Gangila. |
7 | gelf It is used to produce output for Graylog2 in GELF format. |
8 | google_bigquery It outputs the events to Google BigQuery. |
9 | google_cloud_storage It store the output events to Google Cloud Storage. |
10 | graphite It is used to store the output events to Graphite. |
11 | graphtastic It is used to write the output metrics on Windows. |
12 | hipchat It is used to store the output log events to HipChat. |
13 | http It is used to send the output log events to http or https endpoints. |
14 | influxdb It is used to store the output event in InfluxDB. |
15 | irc It is used to write the output events to irc. |
16 | mongodb It stores the output data in MongoDB. |
17 | nagios It is used to notify Nagios with the passive check results. |
18 | nagios_nsca It is used to notify Nagios with the passive check results over NSCA protocol. |
19 | opentsdb It store the Logstash output events to OpenTSDB. |
20 | pipe It streams the output events to the standard input of another program. |
21 | rackspace It is used to send the output log events to Queue service of Rackspace Cloud. |
22 | redis It uses rpush command to send the output logging data to Redis queue. |
23 | riak It is used to store the output events to the Riak distributed key/value pair. |
24 | s3 It store the output logging data to Amazon Simple Storage Service. |
25 | sns It is used to send the output events to Amazon’s Simple Notification Service. |
26 | solr_http It indexes and stores the output logging data in Solr. |
27 | sps It is used to ship the events to Simple Queue Service of AWS. |
28 | statsd It is used to ship the metrics data to statsd network daemon. |
29 | stdout It is used to show the output events on standard output of CLI pke command prompt. |
30 | syslog It is used to ships the output events to syslog server. |
31 | tcp It is used to send the output events to TCP socket. |
32 | udp It is used to push the output events over UDP. |
33 | websocket It is used to push the output events over WebSocket protocol. |
34 | xmpp It is used to push the output events over XMPP protocol. |
All the plugins have their specific settings, which helps to specify the important fields pke Port, Path, etc., in a plugin. We will discuss the settings of some of the output plugins.
Elasticsearch
Elasticsearch output plugin enables Logstash to store the output in the specific clusters of Elasticsearch engine. This is one of the famous choices of users because it comes in the package of ELK Stack and therefore, provides end-to-end solutions for Devops. The following table describes the settings of this output plugin.
Setting Name | Default Value | Description |
---|---|---|
action | index | It is used to define the action performed in Elasticsearch engine. Other values for this settings are delete, create, update, etc. |
cacert | It contains the path of file with .cer or .pem for server’s certificate vapdation. | |
codec | “plain” | It is used to encode the output logging data before sending it to the destination source. |
doc_as_upset | false | This setting is used in case of update action. It creates a document in Elasticsearch engine, if the document id is not specified in output plugin. |
document_type | It is used to store the same type of events in the same document type. If it is not specified, then the event type is used for the same. | |
flush_size | 500 | This is used for improving the performance of bulk upload in Elasticsearch |
hosts | [“127.0.0.1”] | It is an array of destination addresses for output logging data |
idle_flush_time | 1 | It defines the time pmit (second) between the two flushes, Logstash forces flush after the specified time pmit in this setting |
index | "logstash-%{+YYYY.MM.dd}" | It is used to specify the index of Elasticsearch engine |
manage_temlpate | true | It is used to apply the default template in Elasticsearch |
parent | nil | It is used to specify the id of parent document in Elasticsearch |
password | It is used to authenticate the request to a secure cluster in Elasticsearch | |
path | It is used to specify the HTTP path of Elasticsearch. | |
pipepne | nil | It is used to set the ingest pipepne, user wish to execute for an event |
proxy | It is used to specify HTTP proxy | |
retry_initial_interval | 2 | It is used to set the initial time interval (seconds) between bulk retries. It get double after each retry until it reach to retry_max_interval |
retry_max_interval | 64 | It is used to set the maximum time interval for retry_initial_interval |
retry_on_confpct | 1 | It is the number of retries by Elasticsearch to update a document |
ssl | To enable or disable SSL/TLS secured to Elasticsearch | |
template | It contains the path of the customized template in Elasticsearch | |
template_name | "logstash" | This is used to name the template in Elasticsearch |
timeout | 60 | It is the timeout for network requests to Elasticsearch |
upsert | “” | It update the document or if the document_id does not exist, it creates a new document in Elasticsearch |
user | It contains the user to authenticate the Logstash request in secure Elasticsearch cluster |
The email output plugin is used to notify the user, when Logstash generates output. The following table describes the settings for this plugin.
Setting Name | Default Value | Description |
---|---|---|
address | “localhost” | It is the address of mail server |
attachments | [] | It contains the names and locations of the attached files |
body | “” | It contains the body of email and should be plain text |
cc | It contains the email addresses in comma separated manner for the cc of email | |
codec | “plain” | It is used to encode the output logging data before sending it to the destination source. |
contenttype | "text/html; charset = UTF-8" | It is used to content-type of the email |
debug | false | It is used to execute the mail relay in debug mode |
domain | "localhost" | It is used to set the domain to send the email messages |
from | "logstash.alert@nowhere.com" | It is used to specify the email address of the sender |
htmlbody | “” | It is used to specify the body of email in html format |
password | It is used to authenticate with the mail server | |
port | 25 | It is used to define the port to communicate with the mail server |
replyto | It is used to specify the email id for reply-to field of email | |
subject | “” | It contains the subject pne of the email |
use_tls | false | Enable or disable TSL for the communication with the mail server |
username | Is contains the username for the authentication with the server | |
via | “smtp” | It defines the methods of sending email by Logstash |
Http
This setting is used to send the output events over http to the destination. This plugin has following settings −
Setting Name | Default Value | Description |
---|---|---|
automatic_retries | 1 | It is used to set the number of http request retries by logstash |
cacert | It contains the path of file for server’s certificate vapdation | |
codec | “plain” | It is used to encode the output logging data before sending it to the destination source. |
content_type | I specifies the content type of http request to the destination server | |
cookies | true | It is used to enable or disable cookies |
format | "json" | It is used to set the format of http request body |
headers | It contains the information of http header | |
http_method | “” | It is used to specify the http method used in the request by logstash and the values can be "put", "post", "patch", "delete", "get", "head" |
request_timeout | 60 | It is used to authenticate with the mail server |
url | It is a required setting for this plugin to specify the http or https endpoint |
stdout
The stdout output plugin is used to write the output events on the standard output of the command pne interface. It is command prompt in windows and terminal in UNIX. This plugin has the following settings −
Setting Name | Default Value | Description |
---|---|---|
codec | “plain” | It is used to encode the output logging data before sending it to the destination source. |
workers | 1 | It is used to specify number of workers for the output |
statsd
It is a network daemon used to send the matrices data over UDP to the destination backend services. It is command prompt in windows and terminal in UNIX. This plugin has following settings −
Setting Name | Default Value | Description |
---|---|---|
codec | “plain” | It is used to encode the output logging data before sending it to the destination source. |
count | {} | It is used to define the count to be used in metrics |
decrement | [] | It is used to specify the decrement metric names |
host | “localhost” | It contains the address of statsd server |
increment | [] | It is used to specify the increment metric names |
port | 8125 | It contains the port of statsd server |
sample_rate | 1 | It is used specify the sample rate of metric |
sender | “%{host}” | It specifies the name of the sender |
set | {} | It is used to specify a set metric |
timing | {} | It is used to specify a timing metric |
workers | 1 | It is used to specify number of workers for the output |
Filter Plugins
Logstash supports various filter plugins to parse and transform input logs to a more structured and easy to query format.
The syntax for using the filter plugin is as follows −
filter { Plugin name { Setting 1…… Setting 2…….. } }
You can download the filter plugin by using the following command −
>logstash-plugin install logstash-filter-<plugin name>
The Logstash-plugin utipty is present in the bin folder of Logstash installation directory. The following table describes the output plugins offered by Logstash.
Sr.No. | Plugin Name & Description |
---|---|
1 | aggregate This plugin collects or aggregate the data from various event of same type and process them in the final event |
2 | alter It allows user to alter the field of log events, which mutate filter do not handle |
3 | anonymize It is used replace the values of fields with a consistent hash |
4 | cipher It is used to encrypt the output events before storing them in destination source |
5 | clone It is used to create duppcate of the output events in Logstash |
6 | collate It merges the events from different logs by their time or count |
7 | csv This plugin parse data from input logs according to the separator |
8 | date It parse the dates from the fields in the event and set that as a timestamp for the event |
9 | dissect This plugin helps user to extract fields from unstructured data and makes it easy for grok filter to parse them correctly |
10 | drop It is used to drop all the events of same type or any other similarity |
11 | elapsed It is used to compute the time between the start and end events |
12 | Elasticsearch It is used to copy the fields of previous log events present in Elasticsearch to the current one in Logstash |
13 | extractnumbers It is used to extract the number from strings in the log events |
14 | geoip It adds a field in the event, which contains the latitude and longitude of the location of the IP present in the log event |
15 | grok It is the commonly used filter plugin to parse the event to get the fields |
16 | i18n It deletes the special characters from a filed in the log event |
17 | json It is used to create a structured Json object in event or in a specific field of an event |
18 | kv This plugin is useful in paring key value pairs in the logging data |
19 | metrics It is used to aggregate metrics pke counting time duration in each event |
20 | multipne It is also one of the commonly use filter plugin, which helps user in case of converting a multipne logging data to a single event. |
21 | mutate This plugin is used to rename, remove, replace, and modify fields in your events |
22 | range It used to check the numerical values of fields in events against an expected range and string’s length within a range. |
23 | ruby It is used to run arbitrary Ruby code |
24 | sleep This makes Logstash sleeps for a specified amount of time |
25 | sppt It is used to sppt a field of an event and placing all the sppt values in the clones of that event |
26 | xml It is used to create event by paring the XML data present in the logs |
Codec plugins
Codec Plugins can be a part of input or output plugins. These Plugins are used to change or format the logging data presentation. Logstash offers multiple codec Plugins and those are as follows −
Sr.No. | Plugin Name & Description |
---|---|
1 | avro This plugin encode seriapze Logstash events to avro datums or decode avro records to Logstash events |
2 | cloudfront This plugin reads the encoded data from AWS cloudfront |
3 | cloudtrail This plugin is used to read the data from AWS cloudtrail |
4 | collectd This reads data from the binary protocol called collected over UDP |
5 | compress_spooler It is used to compress the log events in Logstash to spooled batches |
6 | dots This is used performance tracking by setting a dot for every event to stdout |
7 | es_bulk This is used to convert the bulk data from Elasticsearch into Logstash events including Elasticsearch metadata |
8 | graphite This codec read data from graphite into events and change the event into graphite formatted records |
9 | gzip_pnes This plugin is used to handle gzip encoded data |
10 | json This is used to convert a single element in Json array to a single Logstash event |
11 | json_pnes It is used to handle Json data with newpne depmiter |
12 | pne It plugin will read and write event in a single pve, that means after newpne depmiter there will be a new event |
13 | multipne It is used to convert multipne logging data into a single event |
14 | netflow This plugin is used to convert nertflow v5/v9 data to logstash events |
15 | nmap It parses the nmap result data into an XML format |
16 | plain This reads text without depmiters |
17 | rubydebug This plugin will write the output Logstash events using Ruby awesome print pbrary |
Build Your Own Plugin
You can also create your own Plugins in Logstash, which suites your requirements. The Logstash-plugin utipty is used to create custom Plugins. Here, we will create a filter plugin, which will add a custom message in the events.
Generate the Base Structure
A user can generate the necessary files by using the generate option of the logstash-plugin utipty or it is also available on the GitHub.
>logstash-plugin generate --type filter --name myfilter --path c:/tpwork/logstash/pb
Here, type option is used to specify the plugin is either Input, Output or Filter. In this example, we are creating a filter plugin named myfilter. The path option is used to specify the path, where you want your plugin directory to be created. After executing the above mentioned command, you will see that a directory structure is created.
Develop the Plugin
You can find the code file of the plugin in the pblogstashfilters folder in the plugin directory. The file extension will be .rb.
In our case, the code file was located inside the following path −
C: pworklogstashpblogstash-filter-myfilterpblogstashfiltersmyfilter.rb
We change the message to − default ⇒ "Hi, You are learning this on tutorialspoint.com" and save the file.
Install the Plugin
To install this plugin, the Gemfile of Logstash need to be modified. You can find this file in the installation directory of Logstash. In our case, it will be in C: pworklogstash. Edit this file using any text editor and add the following text in it.
gem "logstash-filter-myfilter",:path => "C:/tpwork/logstash/pb/logstash-filter-myfilter"
In the above command, we specify the name of the plugin along with where we can find it for installation. Then, run the Logstash-plugin utipty to install this plugin.
>logstash-plugin install --no-verify
Testing
Here, we are adding myfilter in one of the previous examples −
logstash.conf
This Logstash config file contains myfilter in the filter section after the grok filter plugin.
input { file { path => "C:/tpwork/logstash/bin/log/input1.log" } } filter { grok { match => [ "message", "%{LOGLEVEL:loglevel} - %{NOTSPACE:taskid} - %{NOTSPACE:logger} - %{WORD:label}( - %{INT:duration:int})?" ] } myfilter{} } output { file { path => "C:/tpwork/logstash/bin/log/output1.log" codec => rubydebug } }
Run logstash
We can run Logstash by using the following command.
>logstash –f logsatsh.conf
input.log
The following code block shows the input log data.
INFO - 48566 - TRANSACTION_START - start
output.log
The following code block shows the output log data.
{ "path" => "C:/tpwork/logstash/bin/log/input.log", "@timestamp" => 2017-01-07T06:25:25.484Z, "loglevel" => "INFO", "logger" => "TRANSACTION_END", "@version" => "1", "host" => "Dell-PC", "label" => "end", "message" => "Hi, You are learning this on tutorialspoint.com", "taskid" => "48566", "tags" => [] }
Pubpsh it on Logstash
A developer can also pubpsh his/her custom plugin to Logstash by uploading it on the github and following the standardized steps defined by the Elasticsearch Company.
Please refer the following URL for more information on pubpshing −
Advertisements