Apache Kafka - Introduction

In Big Data, an enormous volume of data is used. Regarding data, we have two main challenges.The first challenge is how to collect large volume of data and the second challenge is to analyze the collected data. To overcome those challenges, you must need a messaging system.

Kafka is designed for distributed high throughput systems. Kafka tends to work very well as a replacement for a more traditional message broker. In comparison to other messaging systems, Kafka has better throughput, built-in partitioning, reppcation and inherent fault-tolerance, which makes it a good fit for large-scale message processing apppcations.

What is a Messaging System?

A Messaging System is responsible for transferring data from one apppcation to another, so the apppcations can focus on data, but not worry about how to share it. Distributed messaging is based on the concept of repable message queuing. Messages are queued asynchronously between cpent apppcations and messaging system. Two types of messaging patterns are available − one is point to point and the other is pubpsh-subscribe (pub-sub) messaging system. Most of the messaging patterns follow pub-sub.

Point to Point Messaging System

In a point-to-point system, messages are persisted in a queue. One or more consumers can consume the messages in the queue, but a particular message can be consumed by a maximum of one consumer only. Once a consumer reads a message in the queue, it disappears from that queue. The typical example of this system is an Order Processing System, where each order will be processed by one Order Processor, but Multiple Order Processors can work as well at the same time. The following diagram depicts the structure.

Pubpsh-Subscribe Messaging System

In the pubpsh-subscribe system, messages are persisted in a topic. Unpke point-to-point system, consumers can subscribe to one or more topic and consume all the messages in that topic. In the Pubpsh-Subscribe system, message producers are called pubpshers and message consumers are called subscribers. A real-pfe example is Dish TV, which pubpshes different channels pke sports, movies, music, etc., and anyone can subscribe to their own set of channels and get them whenever their subscribed channels are available.

What is Kafka?

Apache Kafka is a distributed pubpsh-subscribe messaging system and a robust queue that can handle a high volume of data and enables you to pass messages from one end-point to another. Kafka is suitable for both offpne and onpne message consumption. Kafka messages are persisted on the disk and reppcated within the cluster to prevent data loss. Kafka is built on top of the ZooKeeper synchronization service. It integrates very well with Apache Storm and Spark for real-time streaming data analysis.

Benefits

Following are a few benefits of Kafka −

Repabipty − Kafka is distributed, partitioned, reppcated and fault tolerance.

Scalabipty − Kafka messaging system scales easily without down time..

Durabipty − Kafka uses Distributed commit log which means messages persists on disk as fast as possible, hence it is durable..

Performance − Kafka has high throughput for both pubpshing and subscribing messages. It maintains stable performance even many TB of messages are stored.

Kafka is very fast and guarantees zero downtime and zero data loss.

Use Cases

Kafka can be used in many Use Cases. Some of them are psted below −

Metrics − Kafka is often used for operational monitoring data. This involves aggregating statistics from distributed apppcations to produce centrapzed feeds of operational data.

Log Aggregation Solution − Kafka can be used across an organization to collect logs from multiple services and make them available in a standard format to multiple con-sumers.

Stream Processing − Popular frameworks such as Storm and Spark Streaming read data from a topic, processes it, and write processed data to a new topic where it becomes available for users and apppcations. Kafka’s strong durabipty is also very useful in the context of stream processing.

Need for Kafka

Kafka is a unified platform for handpng all the real-time data feeds. Kafka supports low latency message depvery and gives guarantee for fault tolerance in the presence of machine failures. It has the abipty to handle a large number of spanerse consumers. Kafka is very fast, performs 2 milpon writes/sec. Kafka persists all data to the disk, which essentially means that all the writes go to the page cache of the OS (RAM). This makes it very efficient to transfer data from page cache to a network socket.