Skip to content

NimTechnology

Trình bày các công nghệ CLOUD một cách dễ hiểu.

  • Kubernetes & Container
    • Docker
    • Kubernetes
      • Ingress
    • Helm Chart
    • Isito-EnvoyFilter
    • Apache Kafka
      • Kafka
      • Kafka Connect
      • Lenses
    • Vault
    • Longhorn – Storage
    • VictoriaMetrics
  • CI/CD
    • ArgoCD
    • ArgoWorkflows
    • Spinnaker
    • Jenkins
  • Coding
    • Terraform
      • GCP – Google Cloud
      • AWS – Amazon Web Service
    • Golang
    • Laravel
    • Jquery & JavaScript
    • Git
    • Selenium
  • Log & Monitor
    • Prometheus
    • Grafana
    • ELK
      • Kibana
      • Logstash
  • BareMetal
  • Toggle search form

[Kafka] Kafka Topics CLI

Posted on January 31, 2022February 27, 2022 By nim No Comments on [Kafka] Kafka Topics CLI

Ở bài này mình đi tìm hiểu về Kafka CLI, để chạy được kafka bạn tham khảo bài này nhé:

[Kafka-Zookeeper] Starting Kafka and Zookeeper

Contents

  • 1) kafka-topics
    • 1.1) what is the topic in Kfaka?
    • 1.2) what is the offset in Kfaka?
    • 1.3) Practice
      • 1.3.1) create topic
    • 1.4) Brokers
    • 1.5) Topic replication factor
      • 1.5.1) Concept of Leader for a Partition.
    • 1.3) Practice
      • 1.3.2) create topic with partitions and replication-factor.
      • 1.3.3) list topic
      • 1.3.4) show describe of a topic.
      • 1.3.5) delete topic

1) kafka-topics

kafka-topics.sh
thấy lỗi vì chúng ra khai báo thiếu.

1.1) what is the topic in Kfaka?

Vậy Topic trong kafka là gì?

  • Topics: a particular stream of data
    • Similar to a table in a database (without all the constraints)
    • You can have as many topics as you Want
    • A topic is identified by its name.
  • Topics are split in partitions
    • Each partition is ordered
    • Each message within a partition gets an incremental id, called offset

1.2) what is the offset in Kfaka?

vậy tiếp theo offset là gì?

Offset only have a meaning for a specific partition.
+Eg offset 3 in partition 0 doesn’t represent the same data as offset 3 in partition |
Order is guaranteed only within a partition (not across partitions)
Data is kept only for a limited time (default is one week)
Once the data is written to a partition, it can’t be changed (immutability)
Data is assigned randomly to a partition unless a key is provided (more on this later)

Topic example: truck_gps

  • Say you have a fleet of trucks, each truck reports its GPS position to Kafka.
  • You can have a topic trucks_gps that contains the position of all trucks.
  • Each truck will send a message to Kafka every 20 seconds, each message
    will contain the truck ID and the truck position (latitude and longitude)
  • We choose to create that topic with 10 partitions (arbitrary number)

Giờ chúng ta tạo 1 topic:

1.3) Practice

1.3.1) create topic

kafka-topics.sh --bootstrap-server localhost:9092 -topic first-topic --create --partitions 3

ở đây chúng ta tạo 1 topic với 3 partitions.

Chúng ta còn command khác đó là:

kafka-topics.sh --bootstrap-server localhost:9092 -topic first-topic-repl --create --partitions 3 --replication-factor 2

nhưng chúng ta sẽ nhận về 1 lỗi.

Error while executing topic command : Replication factor: 2 larger than available brokers: 1.

giờ chúng ta tiếp tục tìm hiểu: replication-factor và brokers là gì?

1.4) Brokers

  • A Kafka cluster is composed of multiple brokers (servers)
  • Each broker is identified with its ID (integer)
  • Each broker contains certain topic partitions
  • After connecting to any broker (called a bootstrap broker), you will be
    connected to the entire cluster
  • A good number to get started is 3 brokers, bụt some big clusters have over
    100 brokers
  • In these examples we choose to number brokers starting at 100 (arbitrary)

Ví du:
– Topic-A được tạo với 3 partition
– Topic-B được tạo với 2 partition

ta thấy rằng data đã được distributed(phân tán) và con Broker 103 thì không có bất cứ data nào của topic B.

1.5) Topic replication factor

  • Topics should have a replication factor > l (usually between 2 and 3)
  • This way ifa broker is down, another broker can serve the data
  • Example:Topic-A with 2 partitions and replication factor of 2

Ví dụ. nếu chúng ta có lost Broker 102 thì broker 101 và 103 vẫn có thể phụ vụ được.

1.5.1) Concept of Leader for a Partition.

• At any time only One Broker can be a leader for a given partition.
• Only that leader can receive and serve data for a partition

• The other brokers will synchronize the data
• Therefore each partition has one leader and multiple ISR (in-sync replica)

sau khi tìm hiểu 1 mớ lý thuyết thì bạn sẽ thấy là starting Kafka mình chỉ có 1 Broker nên nếu sét –replication-factor 2 thì sẽ bị lỗi

1.3) Practice

1.3.2) create topic with partitions and replication-factor.

kafka-topics.sh --bootstrap-server localhost:9092 -topic first-topic-repl --create --partitions 3 --replication-factor 1

1.3.3) list topic

Giờ là list topics

kafka-topics.sh --bootstrap-server localhost:9092 --list

1.3.4) show describe of a topic.

kafka-topics.sh --bootstrap-server localhost:9092 --topic first-topic --describe

1.3.5) delete topic

xóa 1 topic:

kafka-topics.sh --bootstrap-server localhost:9092 --topic first-topic-repl --delete

Apache Kafka, BareMetal, Kafka

Post navigation

Previous Post: [Kafka-Zookeeper] Starting Kafka and Zookeeper
Next Post: [Kafka] Kafka Console Producer CLI.

More Related Articles

[Kafka-connect] Single Message Transform: lesson 9 – Cast Apache Kafka
[Kafka/Schema-registry] Installing Schema-registry to use for the Kafka and Kafka-connect model Apache Kafka
[Kafka-connect] Streaming the data of MySQL throughs Kafka-connect and Debezium plugin. Apache Kafka
[Kafka] Install kafka and zookeeper cluster on kubernetes. Apache Kafka
[Node exporter] Install node_exporter on MacOS BareMetal
[Kafka-connect] Single Message Transform: lesson 7 – TimeStampRouter and MessageTimestampRouter – Custom format topic name with timestamp Apache Kafka

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Tham Gia Group DevOps nhé!
Để Nim có nhiều động lực ra nhiều bài viết.
Để nhận được những thông báo mới nhất.

Recent Posts

  • [Argocd/Vault] Integrate Vault inside Argocd by the plugin July 1, 2022
  • [Vault] Using Service Acount of Kubernetes to login Vault system. June 28, 2022
  • Protected: My Assignment  June 24, 2022
  • [Spinnaker] Spinnaker writes too many logs – Reduce spinnaker log level June 22, 2022
  • [Jenkins] Jobs will be created automatically by Jenkins Job Builder June 20, 2022

Archives

  • July 2022
  • June 2022
  • May 2022
  • April 2022
  • March 2022
  • February 2022
  • January 2022
  • December 2021
  • November 2021
  • October 2021
  • September 2021
  • August 2021
  • July 2021
  • June 2021

Categories

  • BareMetal
  • CI/CD
    • ArgoCD
    • ArgoWorkflows
    • Jenkins
    • Spinnaker
  • Coding
    • Git
    • Golang
    • Jquery & JavaScript
    • Laravel
    • Selenium
    • Terraform
      • AWS – Amazon Web Service
      • GCP – Google Cloud
  • Kubernetes & Container
    • Apache Kafka
      • Kafka
      • Kafka Connect
      • Lenses
    • Docker
    • Helm Chart
    • Isito-EnvoyFilter
    • Kubernetes
      • Ingress
    • Longhorn – Storage
    • Vault
    • VictoriaMetrics
  • Log & Monitor
    • ELK
      • Kibana
      • Logstash
    • Grafana
    • Prometheus
  • Uncategorized
  • Admin

Copyright © 2022 NimTechnology.