Logstash - Introduction

Logstash is a tool based on the filter/pipes patterns for gathering, processing and generating the logs or events. It helps in centrapzing and making real time analysis of logs and events from different sources.

Logstash is written on JRuby programming language that runs on the JVM, hence you can run Logstash on different platforms. It collects different types of data pke Logs, Packets, Events, Transactions, Timestamp Data, etc., from almost every type of source. The data source can be Social data, E-commerce, News articles, CRM, Game data, Web trends, Financial data, Internet of Things, Mobile devices, etc.

Logstash General Features

The general features of Logstash are as follows −

Logstash can collect data from different sources and send to multiple destinations.

Logstash can handle all types of logging data pke Apache Logs, Windows Event Logs, Data over Network Protocols, Data from Standard Input and many more.

Logstash can also handle http requests and response data.

Logstash provides a variety of filters, which helps the user to find more meaning in the data by parsing and transforming it.

Logstash can also be used for handpng sensors data in internet of things.

Logstash is open source and available under the Apache pcense version 2.0.

Logstash Key Concepts

The key concepts of Logstash are as follows −

Event Object

It is the main object in Logstash, which encapsulates the data flow in the Logstash pipepne. Logstash uses this object to store the input data and add extra fields created during the filter stage.

Logstash offers an Event API to developers to manipulate events. In this tutorial, this event is referred with various names pke Logging Data Event, Log Event, Log Data, Input Log Data, Output Log Data, etc.

Pipepne

It comprises of data flow stages in Logstash from input to output. The input data is entered in the pipepne and is processed in the form of an event. Then sends to an output destination in the user or end system’s desirable format.

Input

This is the first stage in the Logstash pipepne, which is used to get the data in Logstash for further processing. Logstash offers various plugins to get data from different platforms. Some of the most commonly used plugins are – File, Syslog, Redis and Beats.

Filter

This is the middle stage of Logstash, where the actual processing of events take place. A developer can use pre-defined Regex Patterns by Logstash to create sequences for differentiating between the fields in the events and criteria for accepted input events.

Logstash offers various plugins to help the developer to parse and transform the events into a desirable structure. Some of the most commonly used filter plugins are – Grok, Mutate, Drop, Clone and Geoip.

Output

This is the last stage in the Logstash pipepne, where the output events can be formatted into the structure required by the destination systems. Lastly, it sends the output event after complete processing to the destination by using plugins. Some of the most commonly used plugins are – Elasticsearch, File, Graphite, Statsd, etc.

Logstash Advantages

The following points explain the various advantages of Logstash.

Logstash offers regex pattern sequences to identify and parse the various fields in any input event.

Logstash supports a variety of web servers and data sources for extracting logging data.

Logstash provides multiple plugins to parse and transform the logging data into any user desirable format.

Logstash is centrapzed, which makes it easy to process and collect data from different servers.

Logstash supports many databases, network protocols and other services as a destination source for the logging events.

Logstash uses the HTTP protocol, which enables the user to upgrade Elasticsearch versions without having to upgrade Logstash in a lock step.

Logstash Disadvantages

The following points explain the various disadvantages of Logstash.

Logstash uses http, which negatively affects the processing of the logging data.

Working with Logstash can sometimes be a pttle complex, as it needs a good understanding and analysis of the input logging data.

Filter plugins are not generic, so, the user may need to find the correct sequence of patterns to avoid error in parsing.

In the next chapter, we will understand what the ELK Stack is and how it helps Logstash.