How to code a tool to connect Kafka by Golang

Contents

1) Overview

Bạn cần viết 1 application để connect vào kafka bằng Golang.

Chúng ta có 2 lựa chọn:
https://github.com/confluentinc/confluent-kafka-go
https://github.com/IBM/sarama

Hãy tìm hiểu về sơ về cách thức hoạt động và ưu nhược điểm của 2 package trên.

`confluent-kafka-go`

Key Features:

High Performance: It’s a lightweight wrapper around librdkafka, a C client known for its performance.
Reliability: Leverages the work done in librdkafka across multiple clients.
Support: Offers commercial support through Confluent.
Future Proof: Developed by the creators of Kafka, ensuring it keeps pace with Apache Kafka and Confluent Platform developments.

`sarama`

Key Features:

API Documentation: Offers detailed API documentation and examples.
Testing Support: Includes mocks for testing in the mocks subpackage.
Compatibility and Stability: Provides a “2 releases + 2 months” compatibility guarantee for Kafka and Go versions.
Contributing: Detailed contribution guidelines and technical information available on its wiki.

Comparison and Considerations

Performance vs. Ease of Use: confluent-kafka-go might offer better performance due to its direct reliance on librdkafka, whereas sarama could be more flexible or easier to use, depending on its API and implementation.
Support and Reliability: confluent-kafka-go has the backing of Confluent, which might offer better reliability and support, especially in a commercial context.
Community and Documentation: While both have good documentation, sarama seems to emphasize community involvement and extensive documentation, including a FAQ section and testing support.

Hoặc bạn có thể tham khảo link này:
https://www.reddit.com/r/golang/comments/olrd34/confluentkafkago_or_shopifysarama/

Example code.

`confluent-kafka-go`

package main

import (
    "fmt"
    "github.com/confluentinc/confluent-kafka-go/v2/kafka"
)

func main() {

    p, err := kafka.NewProducer(&kafka.ConfigMap{"bootstrap.servers": "<HOST or IP:PORT>"})
    if err != nil {
        panic(err)
    }

    defer p.Close()

    // Delivery report handler for produced messages
    go func() {
        for e := range p.Events() {
            switch ev := e.(type) {
            case *kafka.Message:
                if ev.TopicPartition.Error != nil {
                    fmt.Printf("Delivery failed: %v\n", ev.TopicPartition)
                } else {
                    fmt.Printf("Delivered message to %v\n", ev.TopicPartition)
                }
            }
        }
    }()

    // Produce messages to topic (asynchronously)
    topic := "<Topic-Name>"
    for _, word := range []string{"Welcome", "to", "the", "Confluent", "Kafka", "Golang", "client"} {
        p.Produce(&kafka.Message{
            TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: kafka.PartitionAny},
            Value:          []byte(word),
        }, nil)
    }

    // Wait for message deliveries before shutting down
    p.Flush(15 * 1000)
}

In this example:

Replace "HOST or IP:PORT" with the address of your Kafka broker.
Replace "Topic-Name" with the name of the Kafka topic you want to send the message to.
The messages "Welcome", "to", "the", "Confluent", "Kafka", "Golang", "client" is sent to the Kafka topic. You can replace this with any message you wish to send.

Dockerfile

It’s a lightweight wrapper around librdkafka, a C client known for its performance.
nên là việc Dockerfile cũng có nhiều các đặt biệt.
https://stackoverflow.com/questions/76177554/how-can-i-fix-my-docker-build-failing-while-my-go-build-succeeds-dockerfile-inc (Bạn nên đọc phần update)

FROM golang:1.21.5-alpine

# Update and install necessary packages
RUN apk update && apk add git
RUN apk add --no-progress --no-cache gcc musl-dev

# Set environment variables
ENV CGO_ENABLED=1
ENV GO111MODULE=on
ENV GOPATH /go
ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH

# Create and set working directory
RUN mkdir -p "$GOPATH/src" "$GOPATH/bin" && chmod -R 777 "$GOPATH"
WORKDIR $GOPATH/src/devops

# Copy the source code
COPY . .

# Download dependencies
RUN go mod download

# Build the application
RUN GOOS=linux go build -tags musl -ldflags '-extldflags "-static"' -o app

# Set the entry point for the container
ENTRYPOINT ["./app"]

# Expose the port the app runs on
EXPOSE 1994

`sarama`

package main

import (
    "log"

    "github.com/IBM/sarama"
)

func main() {
    // Kafka broker configuration
    brokers := []string{"localhost:9092"} // replace with your Kafka broker addresses

    // Create a new Sarama config instance
    config := sarama.NewConfig()
    config.Producer.Return.Successes = true

    // Create a new producer using the given broker list and config
    producer, err := sarama.NewSyncProducer(brokers, config)
    if err != nil {
        log.Fatalf("Failed to start Sarama producer: %s", err)
    }
    defer producer.Close()

    // Define the message to send
    topic := "your_topic" // replace with your Kafka topic
    message := &sarama.ProducerMessage{
        Topic: topic,
        Value: sarama.StringEncoder("Hello, Kafka!"),
    }

    // Send the message
    partition, offset, err := producer.SendMessage(message)
    if err != nil {
        log.Fatalf("Failed to send message: %s", err)
    }

    log.Printf("Message sent to partition %d at offset %d\n", partition, offset)
}

In this example:

Replace "localhost:9092" with the address of your Kafka broker.
Replace "your_topic" with the name of the Kafka topic you want to send the message to.
The message "Hello, Kafka!" is sent to the Kafka topic. You can replace this with any message you wish to send.