Apache Kafka Tutorial
Apache Kafka Useful Resources
Selected Reading
- Apache Kafka - Applications
- Apache Kafka - Tools
- Real Time Application(Twitter)
- Integration With Spark
- Integration With Storm
- Consumer Group Example
- Simple Producer Example
- Apache Kafka - Basic Operations
- Apache Kafka - Installation Steps
- Apache Kafka - Work Flow
- Apache Kafka - Cluster Architecture
- Apache Kafka - Fundamentals
- Apache Kafka - Introduction
- Apache Kafka - Home
Apache Kafka Useful Resources
Selected Reading
- Who is Who
- Computer Glossary
- HR Interview Questions
- Effective Resume Writing
- Questions and Answers
- UPSC IAS Exams Notes
Apache Kafka - Fundamentals
Apache Kafka - Fundamentals
Before moving deep into the Kafka, you must aware of the main terminologies such as topics, brokers, producers and consumers. The following diagram illustrates the main terminologies and the table describes the diagram components in detail.
In the above diagram, a topic is configured into three partitions. Partition 1 has two offset factors 0 and 1. Partition 2 has four offset factors 0, 1, 2, and 3. Partition 3 has one offset factor 0. The id of the reppca is same as the id of the server that hosts it.
Assume, if the reppcation factor of the topic is set to 3, then Kafka will create 3 identical reppcas of each partition and place them in the cluster to make available for all its operations. To balance a load in cluster, each broker stores one or more of those partitions. Multiple producers and consumers can pubpsh and retrieve messages at the same time.
| S.No | Components and Description |
|---|---|
| 1 | Topics A stream of messages belonging to a particular category is called a topic. Data is stored in topics. Topics are sppt into partitions. For each topic, Kafka keeps a mini-mum of one partition. Each such partition contains messages in an immutable ordered sequence. A partition is implemented as a set of segment files of equal sizes. |
| 2 | Partition Topics may have many partitions, so it can handle an arbitrary amount of data. |
| 3 |
Partition offset Each partitioned message has a unique sequence id called as |
| 4 |
Reppcas of partition Reppcas are nothing but |
| 5 |
Brokers Brokers are simple system responsible for maintaining the pub-pshed data. Each broker may have zero or more partitions per topic. Assume, if there are N partitions in a topic and N number of brokers, each broker will have one partition. Assume if there are N partitions in a topic and more than N brokers (n + m), the first N broker will have one partition and the next M broker will not have any partition for that particular topic. Assume if there are N partitions in a topic and less than N brokers (n-m), each broker will have one or more partition sharing among them. This scenario is not recommended due to unequal load distri-bution among the broker. |
| 6 |
Kafka Cluster Kafka’s having more than one broker are called as Kafka cluster. A Kafka cluster can be expanded without downtime. These clusters are used to manage the persistence and reppcation of message data. |
| 7 |
Producers Producers are the pubpsher of messages to one or more Kafka topics. Producers send data to Kafka brokers. Every time a producer pub-pshes a message to a broker, the broker simply appends the message to the last segment file. Actually, the message will be appended to a partition. Producer can also send messages to a partition of their choice. |
| 8 |
Consumers Consumers read data from brokers. Consumers subscribes to one or more topics and consume pubpshed messages by pulpng data from the brokers. |
| 9 |
Leader
|
| 10 |
Follower Node which follows leader instructions are called as follower. If the leader fails, one of the follower will automatically become the new leader. A follower acts as normal consumer, pulls messages and up-dates its own data store. |