[Prometheus] filter or allowlist metrics before sending to the remote storage via remote_write in Prometheus

There are a few options to filter or allowlist metrics before they are sent to the remote storage via remote_write in Prometheus:

1. Use metric relabeling

You can configure relabelling rules on the remote_write config to filter or transform metrics before they are sent. For example:

with metrics:
my_metric_http{status_code=200, cluster=nim] 100
my_metric_http{status_code=500, cluster=nim] 100
k8s_metric_memory{container=web, cluster=nim] 100

remote_write:
  - url: http://remote-storage:9201/write
    write_relabel_configs:
      - source_labels: [__name__]
        regex: ^my_metric_.*
        action: keep

This would only keep metrics matching the regex ^my_metric_.* and drop all others.
We will have the metrics:
my_metric_http{status_code=200, cluster=nim] 100
my_metric_http{status_code=500, cluster=nim] 100

Continuously, you can filter metrics based on the labels in side metrics

            remoteWrite:
              - url: http://remote-storage:9201/write
                name: 'staging-engines'
                remote_timeout: 120s
                bearer_token_file: /etc/secrets/bearer-token-coralogix.txt
                write_relabel_configs:
                  - source_labels: ["status_code"]
                    regex: "^200$"
                    action: keep

Through the above filter. the metrics will be approved such as:
my_metric_http{status_code=200, cluster=nim] 100

2. Use PromQL recording rules

You can create recording rules that match your allowlist metrics and output them to new metrics. Then scrape those rules instead of the original metrics.

1. Create a separate rules file for your allowlist, for example allowlist_rules.yml
2. Define a recording rule for each metric you want to allowlist. The rule should match the original metric name and output it to a new metric name prefixed with something like allowlisted_. For example:

groups:
- name: allowlist
  rules:
  - record: allowlisted_http_requests_total
    expr: http_requests_total

  - record: allowlisted_api_latency_seconds
    expr: api_latency_seconds

3. In prometheus.yml configuration, load this new rules file:

rule_files:
  - "/path/to/allowlist_rules.yml"

4. Configure your remote_write target to only scrape the Prometheus /api/v1/rules endpoint rather than /api/v1/metrics. This ensures it will only get the allowlisted metrics generated from the recording rules. For example:

remote_write:
  - url: https://remote-storage/write
    basic_auth:
      username: prom
      password: rules
    scrape_configs:
      - job_name: 'allowlisted'
        metrics_path: /api/v1/rules

5. Reload Prometheus configuration for the rules and remote_write to take effect.