Ở bài này mình đi tìm hiểu về Kafka CLI, để chạy được kafka bạn tham khảo bài này nhé:
1) kafka-topics
kafka-topics.sh
1.1) what is the topic in Kfaka?
Vậy Topic trong kafka là gì?
- Topics: a particular stream of data
- Similar to a table in a database (without all the constraints)
- You can have as many topics as you Want
- A topic is identified by its name.
- Topics are split in partitions
- Each partition is ordered
- Each message within a partition gets an incremental id, called offset
1.2) what is the offset in Kfaka?
vậy tiếp theo offset là gì?
Offset only have a meaning for a specific partition.
+Eg offset 3 in partition 0 doesn’t represent the same data as offset 3 in partition |
Order is guaranteed only within a partition (not across partitions)
Data is kept only for a limited time (default is one week)
Once the data is written to a partition, it can’t be changed (immutability)
Data is assigned randomly to a partition unless a key is provided (more on this later)
Topic example: truck_gps
- Say you have a fleet of trucks, each truck reports its GPS position to Kafka.
- You can have a topic trucks_gps that contains the position of all trucks.
- Each truck will send a message to Kafka every 20 seconds, each message
will contain the truck ID and the truck position (latitude and longitude) - We choose to create that topic with 10 partitions (arbitrary number)
Giờ chúng ta tạo 1 topic:
1.3) Practice
1.3.1) create topic
kafka-topics.sh --bootstrap-server localhost:9092 -topic first-topic --create --partitions 3
ở đây chúng ta tạo 1 topic với 3 partitions.
Chúng ta còn command khác đó là:
kafka-topics.sh --bootstrap-server localhost:9092 -topic first-topic-repl --create --partitions 3 --replication-factor 2
nhưng chúng ta sẽ nhận về 1 lỗi.
Error while executing topic command : Replication factor: 2 larger than available brokers: 1.
giờ chúng ta tiếp tục tìm hiểu: replication-factor và brokers là gì?
1.4) Brokers
- A Kafka cluster is composed of multiple brokers (servers)
- Each broker is identified with its ID (integer)
- Each broker contains certain topic partitions
- After connecting to any broker (called a bootstrap broker), you will be
connected to the entire cluster - A good number to get started is 3 brokers, bụt some big clusters have over
100 brokers - In these examples we choose to number brokers starting at 100 (arbitrary)
Ví du:
– Topic-A được tạo với 3 partition
– Topic-B được tạo với 2 partition
ta thấy rằng data đã được distributed(phân tán) và con Broker 103 thì không có bất cứ data nào của topic B.
1.5) Topic replication factor
- Topics should have a replication factor > l (usually between 2 and 3)
- This way ifa broker is down, another broker can serve the data
- Example:Topic-A with 2 partitions and replication factor of 2
Ví dụ. nếu chúng ta có lost Broker 102 thì broker 101 và 103 vẫn có thể phụ vụ được.
1.5.1) Concept of Leader for a Partition.
• At any time only One Broker can be a leader for a given partition.
• Only that leader can receive and serve data for a partition
• The other brokers will synchronize the data
• Therefore each partition has one leader and multiple ISR (in-sync replica)
sau khi tìm hiểu 1 mớ lý thuyết thì bạn sẽ thấy là starting Kafka mình chỉ có 1 Broker nên nếu sét –replication-factor 2 thì sẽ bị lỗi
1.3) Practice
1.3.2) create topic with partitions and replication-factor.
kafka-topics.sh --bootstrap-server localhost:9092 -topic first-topic-repl --create --partitions 3 --replication-factor 1
1.3.3) list topic
Giờ là list topics
kafka-topics.sh --bootstrap-server localhost:9092 --list
1.3.4) show describe of a topic.
kafka-topics.sh --bootstrap-server localhost:9092 --topic first-topic --describe
1.3.5) delete topic
xóa 1 topic:
kafka-topics.sh --bootstrap-server localhost:9092 --topic first-topic-repl --delete
1.3.6) update or change the configuration of a topic.
Bạn than khảo thêm bài viết dưới đây.
