There are a few options to filter or allowlist metrics before they are sent to the remote storage via remote_write in Prometheus:
1. Use metric relabeling
You can configure relabelling rules on the remote_write config to filter or transform metrics before they are sent. For example:
with metrics: my_metric_http{status_code=200, cluster=nim] 100
my_metric_http{status_code=500, cluster=nim] 100
k8s_metric_memory{container=web, cluster=nim] 100
remote_write: - url: http://remote-storage:9201/write write_relabel_configs: - source_labels: [__name__] regex: ^my_metric_.* action: keep
This would only keep metrics matching the regex ^my_metric_.* and drop all others.
We will have the metrics:my_metric_http{status_code=200, cluster=nim] 100
my_metric_http{status_code=500, cluster=nim] 100
Continuously, you can filter metrics based on the labels in side metrics
remoteWrite: - url: http://remote-storage:9201/write name: 'staging-engines' remote_timeout: 120s bearer_token_file: /etc/secrets/bearer-token-coralogix.txt write_relabel_configs: - source_labels: ["status_code"] regex: "^200$" action: keep
Through the above filter. the metrics will be approved such as:my_metric_http{status_code=200, cluster=nim] 100
2. Use PromQL recording rules
You can create recording rules that match your allowlist metrics and output them to new metrics. Then scrape those rules instead of the original metrics.
- Create a separate rules file for your allowlist, for example allowlist_rules.yml
- Define a recording rule for each metric you want to allowlist. The rule should match the original metric name and output it to a new metric name prefixed with something like allowlisted_. For example:
groups: - name: allowlist rules: - record: allowlisted_http_requests_total expr: http_requests_total - record: allowlisted_api_latency_seconds expr: api_latency_seconds
- In prometheus.yml configuration, load this new rules file:
rule_files: - "/path/to/allowlist_rules.yml"
- Configure your remote_write target to only scrape the Prometheus /api/v1/rules endpoint rather than /api/v1/metrics. This ensures it will only get the allowlisted metrics generated from the recording rules. For example:
remote_write: - url: https://remote-storage/write basic_auth: username: prom password: rules scrape_configs: - job_name: 'allowlisted' metrics_path: /api/v1/rules
- Reload Prometheus configuration for the rules and remote_write to take effect.