Skip to content

NimTechnology

Trình bày các công nghệ CLOUD một cách dễ hiểu.

  • Kubernetes & Container
    • Docker
    • Kubernetes
      • Ingress
    • Helm Chart
    • Isito-EnvoyFilter
    • Apache Kafka
      • Kafka
      • Kafka Connect
      • Lenses
    • Vault
    • Longhorn – Storage
    • VictoriaMetrics
    • MetalLB
    • Kong Gateway
  • CI/CD
    • ArgoCD
    • ArgoWorkflows
    • Spinnaker
    • Jenkins
    • Harbor
    • TeamCity
    • Git
      • Bitbucket
  • Coding
    • Terraform
      • GCP – Google Cloud
      • AWS – Amazon Web Service
    • Golang
    • Laravel
    • Python
    • Jquery & JavaScript
    • Selenium
  • Log & Monitor
    • DataDog
    • Prometheus
    • Grafana
    • ELK
      • Kibana
      • Logstash
  • BareMetal
    • NextCloud
  • Toggle search form

[Kafka] Kafka Topics CLI

Posted on January 31, 2022October 7, 2022 By nim No Comments on [Kafka] Kafka Topics CLI

Ở bài này mình đi tìm hiểu về Kafka CLI, để chạy được kafka bạn tham khảo bài này nhé:

[Kafka-Zookeeper] Starting Kafka and Zookeeper

Contents

  • 1) kafka-topics
    • 1.1) what is the topic in Kfaka?
    • 1.2) what is the offset in Kfaka?
    • 1.3) Practice
      • 1.3.1) create topic
    • 1.4) Brokers
    • 1.5) Topic replication factor
      • 1.5.1) Concept of Leader for a Partition.
    • 1.3) Practice
      • 1.3.2) create topic with partitions and replication-factor.
      • 1.3.3) list topic
      • 1.3.4) show describe of a topic.
      • 1.3.5) delete topic
      • 1.3.6) update or change the configuration of a topic.

1) kafka-topics

kafka-topics.sh
thấy lỗi vì chúng ra khai báo thiếu.

1.1) what is the topic in Kfaka?

Vậy Topic trong kafka là gì?

  • Topics: a particular stream of data
    • Similar to a table in a database (without all the constraints)
    • You can have as many topics as you Want
    • A topic is identified by its name.
  • Topics are split in partitions
    • Each partition is ordered
    • Each message within a partition gets an incremental id, called offset

1.2) what is the offset in Kfaka?

vậy tiếp theo offset là gì?

Offset only have a meaning for a specific partition.
+Eg offset 3 in partition 0 doesn’t represent the same data as offset 3 in partition |
Order is guaranteed only within a partition (not across partitions)
Data is kept only for a limited time (default is one week)
Once the data is written to a partition, it can’t be changed (immutability)
Data is assigned randomly to a partition unless a key is provided (more on this later)

Topic example: truck_gps

  • Say you have a fleet of trucks, each truck reports its GPS position to Kafka.
  • You can have a topic trucks_gps that contains the position of all trucks.
  • Each truck will send a message to Kafka every 20 seconds, each message
    will contain the truck ID and the truck position (latitude and longitude)
  • We choose to create that topic with 10 partitions (arbitrary number)

Giờ chúng ta tạo 1 topic:

1.3) Practice

1.3.1) create topic

kafka-topics.sh --bootstrap-server localhost:9092 -topic first-topic --create --partitions 3

ở đây chúng ta tạo 1 topic với 3 partitions.

Chúng ta còn command khác đó là:

kafka-topics.sh --bootstrap-server localhost:9092 -topic first-topic-repl --create --partitions 3 --replication-factor 2

nhưng chúng ta sẽ nhận về 1 lỗi.

Error while executing topic command : Replication factor: 2 larger than available brokers: 1.

giờ chúng ta tiếp tục tìm hiểu: replication-factor và brokers là gì?

1.4) Brokers

  • A Kafka cluster is composed of multiple brokers (servers)
  • Each broker is identified with its ID (integer)
  • Each broker contains certain topic partitions
  • After connecting to any broker (called a bootstrap broker), you will be
    connected to the entire cluster
  • A good number to get started is 3 brokers, bụt some big clusters have over
    100 brokers
  • In these examples we choose to number brokers starting at 100 (arbitrary)

Ví du:
– Topic-A được tạo với 3 partition
– Topic-B được tạo với 2 partition

ta thấy rằng data đã được distributed(phân tán) và con Broker 103 thì không có bất cứ data nào của topic B.

1.5) Topic replication factor

  • Topics should have a replication factor > l (usually between 2 and 3)
  • This way ifa broker is down, another broker can serve the data
  • Example:Topic-A with 2 partitions and replication factor of 2

Ví dụ. nếu chúng ta có lost Broker 102 thì broker 101 và 103 vẫn có thể phụ vụ được.

1.5.1) Concept of Leader for a Partition.

• At any time only One Broker can be a leader for a given partition.
• Only that leader can receive and serve data for a partition

• The other brokers will synchronize the data
• Therefore each partition has one leader and multiple ISR (in-sync replica)

sau khi tìm hiểu 1 mớ lý thuyết thì bạn sẽ thấy là starting Kafka mình chỉ có 1 Broker nên nếu sét –replication-factor 2 thì sẽ bị lỗi

1.3) Practice

1.3.2) create topic with partitions and replication-factor.

kafka-topics.sh --bootstrap-server localhost:9092 -topic first-topic-repl --create --partitions 3 --replication-factor 1

1.3.3) list topic

Giờ là list topics

kafka-topics.sh --bootstrap-server localhost:9092 --list

1.3.4) show describe of a topic.

kafka-topics.sh --bootstrap-server localhost:9092 --topic first-topic --describe

1.3.5) delete topic

xóa 1 topic:

kafka-topics.sh --bootstrap-server localhost:9092 --topic first-topic-repl --delete

1.3.6) update or change the configuration of a topic.

Bạn than khảo thêm bài viết dưới đây.

How to manage Kafka topics using command-line
Apache Kafka, BareMetal, Kafka

Post navigation

Previous Post: [Kafka-Zookeeper] Starting Kafka and Zookeeper
Next Post: [Kafka] Kafka Console Producer CLI.

More Related Articles

[windows]Shutdown máy tính client bằng Domain controller BareMetal
[rclone] Mount folder in linux with google drive by rclone. So helpful to backup data! BareMetal
[letsencrypt] DST Root CA X3 Expiration (September 2021) BareMetal
[Ubuntu 20.4] You need to note A few problems when Installing Ubuntu. BareMetal
[Kafka/MSK] Install kafka or MSK on aws through terraform. AWS - Amazon Web Service
[Postgresql] Install postgresql client and trying a few command postgresql. BareMetal

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Tham Gia Group DevOps nhé!
Để Nim có nhiều động lực ra nhiều bài viết.
Để nhận được những thông báo mới nhất.

Recent Posts

  • Experiences for IP Addresses Shortage on EKS Clusters March 29, 2023
  • [Talisman] Discover the sensitive information in your code. March 28, 2023
  • [Prometheus/Grafana] Install Prometheus and Grafana on ubuntu. March 27, 2023
  • [Kong Gateway] WebSocket connection failed March 26, 2023
  • [Nextcloud] Can’t download files to have a size bigger than 2Gi on NextCloud – RaspBerry March 24, 2023

Archives

  • March 2023
  • February 2023
  • January 2023
  • December 2022
  • November 2022
  • October 2022
  • September 2022
  • August 2022
  • July 2022
  • June 2022
  • May 2022
  • April 2022
  • March 2022
  • February 2022
  • January 2022
  • December 2021
  • November 2021
  • October 2021
  • September 2021
  • August 2021
  • July 2021
  • June 2021

Categories

  • BareMetal
    • NextCloud
  • CI/CD
    • ArgoCD
    • ArgoWorkflows
    • Git
      • Bitbucket
    • Harbor
    • Jenkins
    • Spinnaker
    • TeamCity
  • Coding
    • Golang
    • Jquery & JavaScript
    • Laravel
    • Python
    • Selenium
    • Terraform
      • AWS – Amazon Web Service
      • GCP – Google Cloud
  • Kubernetes & Container
    • Apache Kafka
      • Kafka
      • Kafka Connect
      • Lenses
    • Docker
    • Helm Chart
    • Isito-EnvoyFilter
    • Kong Gateway
    • Kubernetes
      • Ingress
    • Longhorn – Storage
    • MetalLB
    • Vault
    • VictoriaMetrics
  • Log & Monitor
    • DataDog
    • ELK
      • Kibana
      • Logstash
    • Grafana
    • Prometheus
  • Uncategorized
  • Admin

Copyright © 2023 NimTechnology.