Tracing là hệ thống khá quan trọng khi bạn xây dụng các mô hình micro service.
Bạn sẽ biết service nào của bạn chậm, function nào tốn nhiều thời gian.
1) Install Jaeger Tracing.
Repo URL: https://jaegertracing.github.io/helm-charts
CHART: jaeger:0.69.1
Bạn sẽ chú ý đến 2 deployment quan trọng:
1.1) Jaeger UI
Đây là nơi mà bạn có thể thấy được các SPAN trong tracing.
Đó chính là trong service: jaeger-query
1.2) Jaeger-collector
Đâu là nơi mà bạn send tracing data từ application của bạn vào Jaeger.
Bạn có thể thấy là jaeger public 2 protocols: HTTP and GRPC.
1.3) Testing with Jaeger Tracing.
1.3.1) Using Golang.
Mính sẽ chạy golang để test sem là Jaeger có work không?
https://github.com/open-telemetry/opentelemetry-go/blob/main/example/jaeger/main.go
// Copyright The OpenTelemetry Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Command jaeger is an example program that creates spans
// and uploads to Jaeger.
package main
import (
"context"
"log"
"time"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/exporters/jaeger"
"go.opentelemetry.io/otel/sdk/resource"
tracesdk "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.17.0"
)
const (
service = "trace-demo"
environment = "production"
id = 1
)
// tracerProvider returns an OpenTelemetry TracerProvider configured to use
// the Jaeger exporter that will send spans to the provided url. The returned
// TracerProvider will also use a Resource configured with all the information
// about the application.
func tracerProvider(url string) (*tracesdk.TracerProvider, error) {
// Create the Jaeger exporter
exp, err := jaeger.New(jaeger.WithCollectorEndpoint(jaeger.WithEndpoint(url)))
if err != nil {
return nil, err
}
tp := tracesdk.NewTracerProvider(
// Always be sure to batch in production.
tracesdk.WithBatcher(exp),
// Record information about this application in a Resource.
tracesdk.WithResource(resource.NewWithAttributes(
semconv.SchemaURL,
semconv.ServiceName(service),
attribute.String("environment", environment),
attribute.Int64("ID", id),
)),
)
return tp, nil
}
func main() {
tp, err := tracerProvider("http://localhost:14268/api/traces") // change it
if err != nil {
log.Fatal(err)
}
// Register our TracerProvider as the global so any imported
// instrumentation in the future will default to using it.
otel.SetTracerProvider(tp)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// Cleanly shutdown and flush telemetry when the application exits.
defer func(ctx context.Context) {
// Do not make the application hang when it is shutdown.
ctx, cancel = context.WithTimeout(ctx, time.Second*5)
defer cancel()
if err := tp.Shutdown(ctx); err != nil {
log.Fatal(err)
}
}(ctx)
tr := tp.Tracer("component-main")
ctx, span := tr.Start(ctx, "foo")
defer span.End()
bar(ctx)
}
func bar(ctx context.Context) {
// Use the global TracerProvider.
tr := otel.Tracer("component-bar")
_, span := tr.Start(ctx, "bar")
span.SetAttributes(attribute.Key("testset").String("value"))
defer span.End()
// Do bar...
}
Bạn lên jager kiểm tra lại. như ảnh thì jaeger đã worked.
2) Open Telemetry Collector.
2.1) Look into OpenTelemetry
Open Telemetry Colloctor có thể lấy được metrics. log, tracing data
Ở bài này thì mình chỉ focus vào việc OpenTelemetry nhận tracing data từ app của bạn, tiếp đến là sử lý gì đó mà tiếp đến là gửi về Jaeger
Như hình vẽ ở trên thì bạn sẽ cần tìm hiều về Receivers, Processers, và Exporter trong Opentelemetry.
Receive mình sẽ cấu hình nhận OTLP
Exporter thì sẽ là Jaeger
Processor khá nhiều thì bạn sẽ cần phải tự tìm hiểu
2.2) Install OpenTelemetry on Kubernetes
Mình sẽ sử dụng helm:
REPO URL: https://open-telemetry.github.io/opentelemetry-helm-charts
CHART: opentelemetry-collector:0.57.1
Values Files:
mode: "deployment"
config:
exporters:
jaeger:
endpoint: jaeger-collector.jaeger:14250
tls:
insecure: true
service:
pipelines:
traces:
exporters:
- jaeger
- logging
Và giờ chúng ta soi configmap chút
Step 1 gửi tracing data vào opentelemetry
Step 2: Data sẽ được sử lý bới OPenTelemetry
Step 3: đưa tracing data đến end point Jaeger đã được config sẵn
2.3) Demo with Golang
https://github.com/open-telemetry/opentelemetry-go/blob/main/example/otel-collector/main.go
// Copyright The OpenTelemetry Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Example using OTLP exporters + collector + third-party backends. For
// information about using the exporter, see:
// https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp?tab=doc#example-package-Insecure
package main
import (
"context"
"fmt"
"log"
"os"
"os/signal"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.17.0"
"go.opentelemetry.io/otel/trace"
)
// Initializes an OTLP exporter, and configures the corresponding trace and
// metric providers.
func initProvider() (func(context.Context) error, error) {
ctx := context.Background()
res, err := resource.New(ctx,
resource.WithAttributes(
// the service name used to display traces in backends
semconv.ServiceName("test-service"),
),
)
if err != nil {
return nil, fmt.Errorf("failed to create resource: %w", err)
}
// If the OpenTelemetry Collector is running on a local cluster (minikube or
// microk8s), it should be accessible through the NodePort service at the
// `localhost:30080` endpoint. Otherwise, replace `localhost` with the
// endpoint of your cluster. If you run the app inside k8s, then you can
// probably connect directly to the service through dns.
ctx, cancel := context.WithTimeout(ctx, time.Second)
defer cancel()
conn, err := grpc.DialContext(ctx, "localhost:30080",
// Note the use of insecure transport here. TLS is recommended in production.
grpc.WithTransportCredentials(insecure.NewCredentials()),
grpc.WithBlock(),
)
if err != nil {
return nil, fmt.Errorf("failed to create gRPC connection to collector: %w", err)
}
// Set up a trace exporter
traceExporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithGRPCConn(conn))
if err != nil {
return nil, fmt.Errorf("failed to create trace exporter: %w", err)
}
// Register the trace exporter with a TracerProvider, using a batch
// span processor to aggregate spans before export.
bsp := sdktrace.NewBatchSpanProcessor(traceExporter)
tracerProvider := sdktrace.NewTracerProvider(
sdktrace.WithSampler(sdktrace.AlwaysSample()),
sdktrace.WithResource(res),
sdktrace.WithSpanProcessor(bsp),
)
otel.SetTracerProvider(tracerProvider)
// set global propagator to tracecontext (the default is no-op).
otel.SetTextMapPropagator(propagation.TraceContext{})
// Shutdown will flush any remaining spans and shut down the exporter.
return tracerProvider.Shutdown, nil
}
func main() {
log.Printf("Waiting for connection...")
ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
defer cancel()
shutdown, err := initProvider()
if err != nil {
log.Fatal(err)
}
defer func() {
if err := shutdown(ctx); err != nil {
log.Fatal("failed to shutdown TracerProvider: %w", err)
}
}()
tracer := otel.Tracer("test-tracer")
// Attributes represent additional key-value descriptors that can be bound
// to a metric observer or recorder.
commonAttrs := []attribute.KeyValue{
attribute.String("attrA", "chocolate"),
attribute.String("attrB", "raspberry"),
attribute.String("attrC", "vanilla"),
}
// work begins
ctx, span := tracer.Start(
ctx,
"CollectorExporter-Example",
trace.WithAttributes(commonAttrs...))
defer span.End()
for i := 0; i < 10; i++ {
_, iSpan := tracer.Start(ctx, fmt.Sprintf("Sample-%d", i))
log.Printf("Doing really hard work (%d / 10)\n", i+1)
<-time.After(time.Second)
iSpan.End()
}
log.Printf("Done!")
}
Sau khi mình chạy đoạn code trên thì
