1) Configure the datadog agent to receive the APM data.
Như mình đã nói thì datadog cho chúng ta free.
Và hiện tại mình đang dụng datadog để monitor cluster k8s(home lab), log (stdout). tiếp mình sẽ setup tracing trên datadog.
Enable datadog via helm chart. thì chúng ta sẽ cần phải config helm như sau
>>>>value of helm chart
datadog:
# enable to receive the data APM on agent datadog.
apm:
enabled: true
portEnabled: true
Lúc bạn sẽ thấy datadog agent sẽ có thêm port 8126 và mình cấu hình nodeport để service ở ngoài k8s cũng có thể gửi vào datadog agent.
2) Inject the APM library of datadog into Your code.
2.1) Golang
Mình cũng lấy 1 example của datadog thôi.
https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/go/?tab=containers
package main
import (
httptrace "gopkg.in/DataDog/dd-trace-go.v1/contrib/net/http"
"gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
"net/http"
"os"
)
func main() {
os.Setenv("DD_AGENT_HOST", "192.168.101.41")
os.Setenv("DD_TRACE_AGENT_PORT", "32001")
tracer.Start()
defer tracer.Stop()
// Create a traced mux router
mux := httptrace.NewServeMux()
// Continue using the router as you normally would.
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hello World!"))
})
http.ListenAndServe(":8080", mux)
}
Bạn sẽ thấy là mình đã thêm env để code có thể gửi về endpoint mà mình muốn.
Và khi bạn truy cập vào web golang bạn sẽ thấy trong apm của datadog sẽ hiện thị như trên
Bạn có thể tiếp tục khám phá tracing
Mình thích echo framework:
https://medium.com/qoala-engineering/datadog-apm-tracing-for-golang-service-46a6d5dce61c
2.2) Apply APM datadog for wordpress.
Bạn cũng đang thấy là mình có 1 web wordpress nè.
Giờ mình muốn tracing monitor cho wordpress của mình bằng datadog.
Còn nếu bạn muốn APM bằng Elastic thì theo dõi bài dưới.
Vì mình chạy wordpress docker nên mình sẽ cần exec vào container.
apt install wget -y wget https://github.com/DataDog/dd-trace-php/releases/download/0.86.1/datadog-php-tracer_0.86.1_amd64.deb dpkg -i datadog-php-tracer_0.86.1_amd64.deb
Nếu bạn cài trên debian thì tham kháo bài dưới:
https://www.devopsschool.com/blog/how-to-install-configure-datadog-apm-with-php-based-application-wordpress-in-linux/
Và log nó như bên dưới.
root@7d2d8c677ede:/var/www/html# dpkg -i datadog-php-tracer_0.86.1_amd64.deb
(Reading database ... 17636 files and directories currently installed.)
Preparing to unpack datadog-php-tracer_0.86.1_amd64.deb ...
Unpacking datadog-php-tracer (0.86.1) over (0.86.1) ...
Setting up datadog-php-tracer (0.86.1) ...
### PHP version
PHP 7.4.20 (cli) (built: Jun 28 2021 21:43:51) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
with Zend OPcache v7.4.20, Copyright (c), by Zend Technologies
with ddtrace v0.86.1, Copyright Datadog, by Datadog
### Installing Datadog PHP tracing extension (ddtrace)
###
### Logging /usr/local/bin/php -i to a file
###
Checking for extension: json
Extension 'json' was found.
### Creating ddtrace.ini
###
; ***** DO NOT EDIT THIS FILE *****
; To overwrite the INI settings for this extension, edit
; the INI file in this directory called "ddtrace-custom.ini"
[datadog]
extension=/opt/datadog-php/extensions/ddtrace-20190902.so
datadog.trace.request_init_hook=/opt/datadog-php/dd-trace-sources/bridge/dd_wrap_autoloader.php
### ddtrace.ini created
###
### Linking ddtrace.ini for supported SAPI's
###
### Found SAPI config directory: /usr/local/etc/php/conf.d
### Linking ddtrace.ini to /usr/local/etc/php/conf.d/98-ddtrace.ini
### Linking ddtrace-custom.ini to /usr/local/etc/php/conf.d/99-ddtrace-custom.ini
###
ddtrace
ddtrace
### Extension enabled successfully
Bạn sẽ thấy nó thông báo là
Bạn chúng ta sẽ cần sửa file: /usr/local/etc/php/conf.d/99-ddtrace-custom.ini
Sau một hồi lần mò thì mình đã biết cách để mò cofig.
https://qiita.com/hogehoge789/items/dd8b28bdc243f4785bb7
Bước 1: Kiểm tra ddtrace đã hoạt động hay chưa?
root@7d2d8c677ede:/var/www/html# php -m | grep ddtrace ddtrace ddtrace
Output như trên là ok.
giờ bạn list config của ddtrace
root@7d2d8c677ede:/var/www/html# php --ri=ddtrace
ddtrace
Datadog PHP tracer extension
For help, check out the documentation at https://docs.datadoghq.com/tracing/languages/php/
(c) Datadog 2020
Datadog tracing support => disabled
Version => 0.86.1
DATADOG TRACER CONFIGURATION => {
"date": "2023-04-03T18:14:30Z",
"os_name": "Linux 7d2d8c677ede 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64",
"os_version": "5.4.0-144-generic",
"version": "0.86.1",
"lang": "php",
"lang_version": "7.4.33",
"env": "production",
"enabled": true,
"service": "wordpress",
"enabled_cli": false,
"agent_url": "http:\/\/192.168.101.41:32001",
"debug": false,
"analytics_enabled": true,
"sample_rate": 1,
"sampling_rules": [],
"tags": [],
"service_mapping": [],
"distributed_tracing_enabled": true,
"priority_sampling_enabled": true,
"dd_version": null,
"architecture": "x86_64",
"sapi": "cli",
"datadog.trace.request_init_hook": "\/opt\/datadog-php\/dd-trace-sources\/bridge\/dd_wrap_autoloader.php",
"open_basedir_configured": false,
"uri_fragment_regex": null,
"uri_mapping_incoming": null,
"uri_mapping_outgoing": null,
"auto_flush_enabled": true,
"generate_root_span": true,
"http_client_split_by_domain": false,
"measure_compile_time": true,
"report_hostname_on_root_span": false,
"traced_internal_functions": null,
"auto_prepend_file_configured": false,
"integrations_disabled": "default",
"enabled_from_env": false,
"opcache.file_cache": null
}
Directive => Local Value => Master Value
ddtrace.disable => Off => Off
ddtrace.cgroup_file => /proc/self/cgroup => /proc/self/cgroup
datadog.trace.request_init_hook => /opt/datadog-php/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog-php/dd-trace-sources/bridge/dd_wrap_autoloader.php
ddtrace.request_init_hook => /opt/datadog-php/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog-php/dd-trace-sources/bridge/dd_wrap_autoloader.php
datadog.trace.agent_url => no value => no value
datadog.agent_host => 192.168.101.41 => 192.168.101.41
datadog.dogstatsd_url => no value => no value
datadog.distributed_tracing => On => On
datadog.dogstatsd_port => 8125 => 8125
datadog.env => production => production
datadog.autofinish_spans => Off => Off
datadog.trace.url_as_resource_names_enabled => On => On
datadog.integrations_disabled => default => default
datadog.priority_sampling => On => On
datadog.service => wordpress => wordpress
datadog.service_name => wordpress => wordpress
datadog.service_mapping => no value => no value
datadog.tags => no value => no value
datadog.trace.global_tags => no value => no value
datadog.trace.agent_port => 32001 => 32001
datadog.trace.analytics_enabled => On => On
datadog.trace.auto_flush_enabled => On => On
datadog.trace.cli_enabled => Off => Off
datadog.trace.measure_compile_time => On => On
datadog.trace.debug => Off => Off
datadog.trace.enabled => Off => On
datadog.trace.health_metrics_enabled => Off => Off
datadog.trace.health_metrics_heartbeat_sample_rate => 0.001 => 0.001
datadog.trace.db_client_split_by_instance => Off => Off
datadog.trace.http_client_split_by_domain => Off => Off
datadog.trace.redis_client_split_by_host => Off => Off
datadog.trace.memory_limit => no value => no value
datadog.trace.report_hostname => Off => Off
datadog.trace.resource_uri_fragment_regex => no value => no value
datadog.trace.resource_uri_mapping_incoming => no value => no value
datadog.trace.resource_uri_mapping_outgoing => no value => no value
datadog.trace.resource_uri_query_param_allowed => no value => no value
datadog.trace.http_url_query_param_allowed => * => *
datadog.trace.http_post_data_param_allowed => no value => no value
datadog.trace.rate_limit => 0 => 0
datadog.trace.sample_rate => 1 => 1
datadog.sampling_rate => 1 => 1
datadog.trace.sampling_rules => [] => []
datadog.span_sampling_rules => [] => []
datadog.span_sampling_rules_file => no value => no value
datadog.trace.header_tags => no value => no value
datadog.trace.x_datadog_tags_max_length => 512 => 512
datadog.trace.propagate_service => Off => Off
datadog.trace.propagation_style_extract => tracecontext,Datadog,B3,B3 single header => tracecontext,Datadog,B3,B3 single header
datadog.propagation_style_extract => tracecontext,Datadog,B3,B3 single header => tracecontext,Datadog,B3,B3 single header
datadog.trace.propagation_style_inject => tracecontext,Datadog => tracecontext,Datadog
datadog.propagation_style_inject => tracecontext,Datadog => tracecontext,Datadog
datadog.trace.propagation_style => tracecontext,Datadog => tracecontext,Datadog
datadog.trace.traced_internal_functions => no value => no value
datadog.trace.agent_timeout => 500 => 500
datadog.trace.agent_connect_timeout => 100 => 100
datadog.trace.debug_prng_seed => -1 => -1
datadog.log_backtrace => Off => Off
datadog.trace.generate_root_span => On => On
datadog.trace.spans_limit => 1000 => 1000
datadog.trace.128_bit_traceid_generation_enabled => Off => Off
datadog.trace.agent_max_consecutive_failures => 3 => 3
datadog.trace.agent_attempt_retry_time_msec => 5000 => 5000
datadog.trace.bgs_connect_timeout => 2000 => 2000
datadog.trace.bgs_timeout => 5000 => 5000
datadog.trace.agent_flush_interval => 5000 => 5000
datadog.trace.agent_flush_after_n_requests => 10 => 10
datadog.trace.shutdown_timeout => 5000 => 5000
datadog.trace.startup_logs => On => On
datadog.trace.agent_debug_verbose_curl => Off => Off
datadog.trace.debug_curl_output => Off => Off
datadog.trace.beta_high_memory_pressure_percent => 80 => 80
datadog.trace.warn_legacy_dd_trace => On => On
datadog.trace.retain_thread_capabilities => Off => Off
datadog.version => no value => no value
datadog.trace.obfuscation_query_string_regexp => (?i)(?:p(?:ass)?w(?:or)?d|pass(?:_?phrase)?|secret|(?:api_?|private_?|public_?|access_?|secret_?)key(?:_?id)?|token|consumer_?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)*(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)*(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|bearer(?:\s|%20)+[a-z0-9\._\-]|token(?::|%3A)[a-z0-9]{13}|gh[opsu]_[0-9a-zA-Z]{36}|ey[I-L](?:[\w=-]|%3D)+\.ey[I-L](?:[\w=-]|%3D)+(?:\.(?:[\w.+\/=-]|%3D|%2F|%2B)+)?|[\-]{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY[\-]{5}[^\-]+[\-]{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY|ssh-rsa(?:\s|%20)*(?:[a-z0-9\/\.+]|%2F|%5C|%2B){100,} => (?i)(?:p(?:ass)?w(?:or)?d|pass(?:_?phrase)?|secret|(?:api_?|private_?|public_?|access_?|secret_?)key(?:_?id)?|token|consumer_?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)*(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)*(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|bearer(?:\s|%20)+[a-z0-9\._\-]|token(?::|%3A)[a-z0-9]{13}|gh[opsu]_[0-9a-zA-Z]{36}|ey[I-L](?:[\w=-]|%3D)+\.ey[I-L](?:[\w=-]|%3D)+(?:\.(?:[\w.+\/=-]|%3D|%2F|%2B)+)?|[\-]{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY[\-]{5}[^\-]+[\-]{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY|ssh-rsa(?:\s|%20)*(?:[a-z0-9\/\.+]|%2F|%5C|%2B){100,}
datadog.trace.client_ip_enabled => Off => Off
datadog.trace.client_ip_header => no value => no value
datadog.trace.forked_process => On => On
datadog.trace.hook_limit => 100 => 100
datadog.trace.agent_max_payload_size => 52428800 => 52428800
datadog.trace.agent_stack_initial_size => 131072 => 131072
datadog.trace.agent_stack_backlog => 12 => 12
datadog.trace.propagate_user_id_default => Off => Off
datadog.dbm_propagation_mode => disabled => disabled
datadog.trace.cakephp_enabled => On => On
datadog.trace.cakephp_analytics_enabled => Off => Off
datadog.cakephp_analytics_enabled => Off => Off
datadog.trace.cakephp_analytics_sample_rate => 1 => 1
datadog.cakephp_analytics_sample_rate => 1 => 1
datadog.trace.codeigniter_enabled => On => On
datadog.trace.codeigniter_analytics_enabled => Off => Off
datadog.codeigniter_analytics_enabled => Off => Off
datadog.trace.codeigniter_analytics_sample_rate => 1 => 1
datadog.codeigniter_analytics_sample_rate => 1 => 1
datadog.trace.curl_enabled => On => On
datadog.trace.curl_analytics_enabled => Off => Off
datadog.curl_analytics_enabled => Off => Off
datadog.trace.curl_analytics_sample_rate => 1 => 1
datadog.curl_analytics_sample_rate => 1 => 1
datadog.trace.elasticsearch_enabled => On => On
datadog.trace.elasticsearch_analytics_enabled => Off => Off
datadog.elasticsearch_analytics_enabled => Off => Off
datadog.trace.elasticsearch_analytics_sample_rate => 1 => 1
datadog.elasticsearch_analytics_sample_rate => 1 => 1
datadog.trace.eloquent_enabled => On => On
datadog.trace.eloquent_analytics_enabled => Off => Off
datadog.eloquent_analytics_enabled => Off => Off
datadog.trace.eloquent_analytics_sample_rate => 1 => 1
datadog.eloquent_analytics_sample_rate => 1 => 1
datadog.trace.guzzle_enabled => On => On
datadog.trace.guzzle_analytics_enabled => Off => Off
datadog.guzzle_analytics_enabled => Off => Off
datadog.trace.guzzle_analytics_sample_rate => 1 => 1
datadog.guzzle_analytics_sample_rate => 1 => 1
datadog.trace.laravel_enabled => On => On
datadog.trace.laravel_analytics_enabled => Off => Off
datadog.laravel_analytics_enabled => Off => Off
datadog.trace.laravel_analytics_sample_rate => 1 => 1
datadog.laravel_analytics_sample_rate => 1 => 1
datadog.trace.lumen_enabled => On => On
datadog.trace.lumen_analytics_enabled => Off => Off
datadog.lumen_analytics_enabled => Off => Off
datadog.trace.lumen_analytics_sample_rate => 1 => 1
datadog.lumen_analytics_sample_rate => 1 => 1
datadog.trace.memcache_enabled => On => On
datadog.trace.memcache_analytics_enabled => Off => Off
datadog.memcache_analytics_enabled => Off => Off
datadog.trace.memcache_analytics_sample_rate => 1 => 1
datadog.memcache_analytics_sample_rate => 1 => 1
datadog.trace.memcached_enabled => On => On
datadog.trace.memcached_analytics_enabled => Off => Off
datadog.memcached_analytics_enabled => Off => Off
datadog.trace.memcached_analytics_sample_rate => 1 => 1
datadog.memcached_analytics_sample_rate => 1 => 1
datadog.trace.mongo_enabled => On => On
datadog.trace.mongo_analytics_enabled => Off => Off
datadog.mongo_analytics_enabled => Off => Off
datadog.trace.mongo_analytics_sample_rate => 1 => 1
datadog.mongo_analytics_sample_rate => 1 => 1
datadog.trace.mongodb_enabled => On => On
datadog.trace.mongodb_analytics_enabled => Off => Off
datadog.mongodb_analytics_enabled => Off => Off
datadog.trace.mongodb_analytics_sample_rate => 1 => 1
datadog.mongodb_analytics_sample_rate => 1 => 1
datadog.trace.mysqli_enabled => On => On
datadog.trace.mysqli_analytics_enabled => Off => Off
datadog.mysqli_analytics_enabled => Off => Off
datadog.trace.mysqli_analytics_sample_rate => 1 => 1
datadog.mysqli_analytics_sample_rate => 1 => 1
datadog.trace.nette_enabled => On => On
datadog.trace.nette_analytics_enabled => Off => Off
datadog.nette_analytics_enabled => Off => Off
datadog.trace.nette_analytics_sample_rate => 1 => 1
datadog.nette_analytics_sample_rate => 1 => 1
datadog.trace.pcntl_enabled => On => On
datadog.trace.pcntl_analytics_enabled => Off => Off
datadog.pcntl_analytics_enabled => Off => Off
datadog.trace.pcntl_analytics_sample_rate => 1 => 1
datadog.pcntl_analytics_sample_rate => 1 => 1
datadog.trace.pdo_enabled => On => On
datadog.trace.pdo_analytics_enabled => Off => Off
datadog.pdo_analytics_enabled => Off => Off
datadog.trace.pdo_analytics_sample_rate => 1 => 1
datadog.pdo_analytics_sample_rate => 1 => 1
datadog.trace.phpredis_enabled => On => On
datadog.trace.phpredis_analytics_enabled => Off => Off
datadog.phpredis_analytics_enabled => Off => Off
datadog.trace.phpredis_analytics_sample_rate => 1 => 1
datadog.phpredis_analytics_sample_rate => 1 => 1
datadog.trace.predis_enabled => On => On
datadog.trace.predis_analytics_enabled => Off => Off
datadog.predis_analytics_enabled => Off => Off
datadog.trace.predis_analytics_sample_rate => 1 => 1
datadog.predis_analytics_sample_rate => 1 => 1
datadog.trace.psr18_enabled => On => On
datadog.trace.psr18_analytics_enabled => Off => Off
datadog.psr18_analytics_enabled => Off => Off
datadog.trace.psr18_analytics_sample_rate => 1 => 1
datadog.psr18_analytics_sample_rate => 1 => 1
datadog.trace.roadrunner_enabled => On => On
datadog.trace.roadrunner_analytics_enabled => Off => Off
datadog.roadrunner_analytics_enabled => Off => Off
datadog.trace.roadrunner_analytics_sample_rate => 1 => 1
datadog.roadrunner_analytics_sample_rate => 1 => 1
datadog.trace.slim_enabled => On => On
datadog.trace.slim_analytics_enabled => Off => Off
datadog.slim_analytics_enabled => Off => Off
datadog.trace.slim_analytics_sample_rate => 1 => 1
datadog.slim_analytics_sample_rate => 1 => 1
datadog.trace.symfony_enabled => On => On
datadog.trace.symfony_analytics_enabled => Off => Off
datadog.symfony_analytics_enabled => Off => Off
datadog.trace.symfony_analytics_sample_rate => 1 => 1
datadog.symfony_analytics_sample_rate => 1 => 1
datadog.trace.web_enabled => On => On
datadog.trace.web_analytics_enabled => Off => Off
datadog.web_analytics_enabled => Off => Off
datadog.trace.web_analytics_sample_rate => 1 => 1
datadog.web_analytics_sample_rate => 1 => 1
datadog.trace.wordpress_enabled => On => On
datadog.trace.wordpress_analytics_enabled => Off => Off
datadog.wordpress_analytics_enabled => Off => Off
datadog.trace.wordpress_analytics_sample_rate => 1 => 1
datadog.wordpress_analytics_sample_rate => 1 => 1
datadog.trace.yii_enabled => On => On
datadog.trace.yii_analytics_enabled => Off => Off
datadog.yii_analytics_enabled => Off => Off
datadog.trace.yii_analytics_sample_rate => 1 => 1
datadog.yii_analytics_sample_rate => 1 => 1
datadog.trace.zendframework_enabled => On => On
datadog.trace.zendframework_analytics_enabled => Off => Off
datadog.zendframework_analytics_enabled => Off => Off
datadog.trace.zendframework_analytics_sample_rate => 1 => 1
datadog.zendframework_analytics_sample_rate => 1 => 1
Giờ file config sẽ như sau:
root@7d2d8c677ede:/var/www/html# cat /usr/local/etc/php/conf.d/99-ddtrace-custom.ini [ddtrace] datadog.service=wordpress datadog.service_name=nimtechnology datadog.env=production datadog.agent_host=192.168.101.41 datadog.trace.agent_port=32001 datadog.trace.analytics_enabled=true datadog.trace.auto_flush_enabled=true
Datadog tự lần mò và detect ra các service trên.
