# Introduction

Developed by LinkedIn and open sourced in 2010.

Apache Kafka is a publish/subscribe messaging system designed to solve this problem. It is often described as a “distributed commit log” or, more recently, as a “distributing streaming platform.”&#x20;

A filesystem or database commit log is designed to provide a durable record of all transactions so that they can be replayed to build the state of a system consistently.



### **Messages**

The unit of data within Kafka is called a _message_.  In the DB world, it's similar to a Row.

The message is simply an array of bytes and does not have a specific format or meaning.

### **Topic**

The topic is a category or feed name to which messages are published, like a TV Channel / Radio station.

### Partitions

Topics are additionally broken down into several partitions.

Messages are written to it in an append-only fashion.

Note that as a topic typically has multiple partitions, message time-ordering is not guaranteed across the entire topic, just within a single partition.

Partitions provide redundancy and scalability. The partition can be hosted on a different server.

<figure><img src="https://1471795080-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FrtXPLjVTxuTGIysjCx89%2Fuploads%2FJFH9QurBoyDNckRnOndb%2Fimage.png?alt=media&#x26;token=2fc162f5-3a06-4080-ac93-090141a6539a" alt=""><figcaption><p>Src: Oreilly Kafka Book</p></figcaption></figure>

### **Keys**

&#x20;A message can have an optional bit of metadata, a _key_. As with the message, the key is also a byte array with no specific meaning to Kafka. Keys are used when messages are to be written to partitions in a more controlled manner.&#x20;

Remember, MongoDB data has \_id field. Similarly, keys can be used to relay messages.

TV Shows you have Season/Episode

### **Offset**

Offset values in Kafka are long integers uniquely identifying each message within a partition. They start at 0 for the first message in a partition and increment by 1 for each subsequent message. For example, if a partition has five messages, the offsets would be 0, 1, 2, 3, and 4.

### **Batches**

A _batch_ is a collection of messages on the same topic and partition.

### Producers

_Producers_ create new messages. In general, a message will be produced on a specific topic.

In general, a message will be produced to a specific topic. By default, the producer does not care what partition a particular message is written to and will evenly balance messages over all partitions of a topic.

In some cases, the producer will direct messages to specific partitions. This is typically done using the message key and a partitioner to generate a hash of the key and map it to a particular partition.

### Consumers

Consumers work as part of a _consumer group_, one or more consumers working together to consume a topic. **The group ensures that each partition is only consumed by one member.**

<figure><img src="https://1471795080-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FrtXPLjVTxuTGIysjCx89%2Fuploads%2FfCMbD2oqUtVogF5paEWp%2Fimage.png?alt=media&#x26;token=a9a61391-b04f-4263-801e-b908f72cb556" alt=""><figcaption><p>Src: Oreilly Kafka Book</p></figcaption></figure>

### Broker

A single Kafka server is called a _broker_. The broker receives messages from producers, assigns offsets to them, and commits the messages to storage on disk.&#x20;

Kafka brokers are designed to operate as part of a _cluster_.

One broker will also function as the cluster controller within a cluster of broker&#x73;_._

_A partition may be assigned to multiple brokers, which will result in Replication._



<figure><img src="https://1471795080-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FrtXPLjVTxuTGIysjCx89%2Fuploads%2FLot5uxCOZaxkWYNhgUhv%2Fimage.png?alt=media&#x26;token=02a37108-fbee-462a-a551-b633b914c2b9" alt=""><figcaption><p>Src: Oreilly Kafka Book</p></figcaption></figure>

### **At least Once Delivery**

* Automatic offset commits provide "at least once" delivery semantics because if the consumer fails after processing a message but before the next auto-commit, it may re-process some messages after restarting.
* This behavior ensures no message loss but can lead to duplicate processing.

### At most Delivery

* The message is sent a maximum of only one time.



