[Kafka] Kafka Topics CLI

Ở bài này mình đi tìm hiểu về Kafka CLI, để chạy được kafka bạn tham khảo bài này nhé:

1) kafka-topics

kafka-topics.sh

1.1) what is the topic in Kfaka?

Vậy Topic trong kafka là gì?

• Topics: a particular stream of data
  • Similar to a table in a database (without all the constraints)
  • You can have as many topics as you Want
  • A topic is identified by its name.
• Topics are split in partitions
  • Each partition is ordered
  • Each message within a partition gets an incremental id, called offset

1.2) what is the offset in Kfaka?

vậy tiếp theo offset là gì?

Offset only have a meaning for a specific partition.
+Eg offset 3 in partition 0 doesn’t represent the same data as offset 3 in partition |
Order is guaranteed only within a partition (not across partitions)
Data is kept only for a limited time (default is one week)
Once the data is written to a partition, it can’t be changed (immutability)
Data is assigned randomly to a partition unless a key is provided (more on this later)

Topic example: truck_gps

• Say you have a fleet of trucks, each truck reports its GPS position to Kafka.
• You can have a topic trucks_gps that contains the position of all trucks.
• Each truck will send a message to Kafka every 20 seconds, each message
  will contain the truck ID and the truck position (latitude and longitude)
• We choose to create that topic with 10 partitions (arbitrary number)

Giờ chúng ta tạo 1 topic:

1.3) Practice

1.3.1) create topic

kafka-topics.sh --bootstrap-server localhost:9092 -topic first-topic --create --partitions 3

ở đây chúng ta tạo 1 topic với 3 partitions.

Chúng ta còn command khác đó là:

kafka-topics.sh --bootstrap-server localhost:9092 -topic first-topic-repl --create --partitions 3 --replication-factor 2

nhưng chúng ta sẽ nhận về 1 lỗi.

Error while executing topic command : Replication factor: 2 larger than available brokers: 1.

giờ chúng ta tiếp tục tìm hiểu: replication-factor và brokers là gì?

1.4) Brokers

• A Kafka cluster is composed of multiple brokers (servers)
• Each broker is identified with its ID (integer)
• Each broker contains certain topic partitions
• After connecting to any broker (called a bootstrap broker), you will be
  connected to the entire cluster
• A good number to get started is 3 brokers, bụt some big clusters have over
  100 brokers
• In these examples we choose to number brokers starting at 100 (arbitrary)

Ví du:
– Topic-A được tạo với 3 partition
– Topic-B được tạo với 2 partition

ta thấy rằng data đã được distributed(phân tán) và con Broker 103 thì không có bất cứ data nào của topic B.

1.5) Topic replication factor

• Topics should have a replication factor > l (usually between 2 and 3)
• This way ifa broker is down, another broker can serve the data
• Example:Topic-A with 2 partitions and replication factor of 2

Ví dụ. nếu chúng ta có lost Broker 102 thì broker 101 và 103 vẫn có thể phụ vụ được.

1.5.1) Concept of Leader for a Partition.

• At any time only One Broker can be a leader for a given partition.
• Only that leader can receive and serve data for a partition
• The other brokers will synchronize the data
• Therefore each partition has one leader and multiple ISR (in-sync replica)

sau khi tìm hiểu 1 mớ lý thuyết thì bạn sẽ thấy là starting Kafka mình chỉ có 1 Broker nên nếu sét –replication-factor 2 thì sẽ bị lỗi

1.3) Practice

1.3.2) create topic with partitions and replication-factor.

kafka-topics.sh --bootstrap-server localhost:9092 -topic first-topic-repl --create --partitions 3 --replication-factor 1

1.3.3) list topic

Giờ là list topics

kafka-topics.sh --bootstrap-server localhost:9092 --list

1.3.4) show describe of a topic.

kafka-topics.sh --bootstrap-server localhost:9092 --topic first-topic --describe

1.3.5) delete topic

xóa 1 topic:

kafka-topics.sh --bootstrap-server localhost:9092 --topic first-topic-repl --delete